Systems and methods for acquiring ppg signals for measuring blood pressure

ABSTRACT

Systems and methods for acquiring photoplethysmographic (PPG) signals for measuring blood pressure can include a computing device acquiring a sequence of images representing transdermal optical data of a subject, and generating a corresponding sequence of downsampled color frames. The computing device can identify, in each downsampled color frame, a respective central image block representing a central image region of the downsampled color frame and having a first size smaller than a second size of the downsampled color frame. The computing device can generate, for each downsampled color frame, a corresponding color intensity value based on the respective central image block. The computing device can generate, using color intensity values corresponding to the sequence of downsampled color frames, a PPG signal to determine a blood pressure value of the subject.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.17/511,548 filed on Oct. 27, 2021, and entitled “SYSTEMS AND METHODS FORACQUIRING PPG SIGNALS FOR MEASURING BLOOD PRESSURE,” which claimspriority to, and the benefit of, U.S. Provisional Application No.63/107,927 filed on Oct. 30, 2020, and entitled “DEVICES AND METHODS FORBLOOD PRESSURE ESTIMATION USING TRANSDERMAL OPTICAL RECORDING,” and U.S.Provisional Application No. 63/220,369 filed on Jul. 9, 2021, andentitled “DEVICES AND METHODS FOR BLOOD PRESSURE ESTIMATION USINGTRANSDERMAL OPTICAL RECORDING,” all of these applications areincorporated herein by reference in their entirety.

FIELD OF THE DISCLOSURE

The present application relates generally to devices and methods forblood pressure estimation using transdermal optical recordings.Specifically, the present application relates to novel techniques ofmeasuring blood pressure using computing devices equipped with a camera.

BACKGROUND

Hypertension is a major challenge in nowadays world. This silent diseasebelongs to the non-communicable diseases class (NCD) that are the causeof more than 50% total annual deaths. It alone is the cause of around7.5 million deaths world-wide. The most dangerous aspect of this diseaseis its high prevalence in population. That is, about 1.13 billion peoplesuffer from hypertension, and it spreads all around the world.

A too high blood pressure is known to be an important risk factor of ahuge amount of cardiovascular diseases such as stroke or coronary heartdisease. However hypertension is a disease without any apparent symptombefore a crisis occur which make of it a very dangerous and treacherousdisease. According to World Health Organization (WHO) less than onepeople out of five that have hypertension has it under control.

Since no apparent symptom is visible before an accident occur, the onlyway of being aware someone suffers from hypertension is to measure it.Blood pressure (BP) is measured thanks to two numbers: systolic bloodpressure (SBP) which is the rise of pressure caused by a heartbeat anddiastolic blood pressure (DBP) corresponding to the following decreasein BP caused by the ventricles loading blood in order to prepare thenext beat. They are measured in mmHg and often transcribed by SBP/DBP. Anormal blood pressure lies between 110/70 mmHg and 120/80 mmHg. Above115/75 mmHg, cardiovascular risk double at each increment of 20/10 mmHg.

SUMMARY

According to one aspect, a method can include acquiring, by aphotodetector of a computing device, a sequence of images representingtransdermal optical data of a subject. The method can includegenerating, by the computing device, a sequence of downsampled colorframes corresponding to the sequence of images by downsampling arespective color frame for each image of the sequence of images. Themethod can include identifying, by the computing device, in eachdownsampled color frame of the sequence of downsampled color frames, arespective image block representing a central image region of thedownsampled color frame and having a first size smaller than a secondsize of the downsampled color frame. The method can include generating,by the computing device, for each downsampled color frame of thesequence of downsampled color frames, a corresponding color intensityvalue based on the respective image block representing a central imageregion of the downsampled color frame. The method can includegenerating, by the computing device, using color intensity valuescorresponding to the sequence of downsampled color frames, aphotoplethysmographic (PPG) signal of the subject to determine a bloodpressure value of the subject.

In some implementations, the method can further include the computingdevice determining a blood pressure value of the subject using the PPGsignal, and displaying the blood pressure value of the user on a displaydevice. Determining the blood pressure value of the subject can includeusing an optical blood pressure monitoring (OBPM) algorithm. In someimplementations, the respective color frame can be a green color frameor a red color frame. In some implementations, each downsampled colorframe can have an n by n size and the respective image blockrepresenting the central image region of the downsampled color frame canhave an m by m size where n and m are integers and m is smaller than n.The integer n can be equal to 5 and m can be equal to 3.

In some implementations, the PPG signal can be a color intensity signalrepresenting average color intensity values of image blocks of thesequence of downsampled color frames. The method can further includefiltering the PPG signal using a high-pass filter, and determining theblood pressure measurement of the subject using a high-pass filteredversion of the PPG signal. The high-pass filter can have a cut-offfrequency between 0.4 Hz and 4 Hz. The method can further includedetermining a measurement of a pulse of the subject using the PPGsignal.

According to one other aspect, a computing device can include aphotodetector, a processor and a memory storing computer codeinstructions. The computer code instructions when executed by theprocessor can cause the photodetector to acquire a sequence of imagesrepresenting transdermal optical data of a subject. The processor cangenerate a sequence of downsampled color frames corresponding to thesequence of images by downsampling a respective color frame for eachimage of the sequence of images. The processor can identify, in eachdownsampled color frame of the sequence of downsampled color frames, arespective image block representing a central image region of thedownsampled color frame and having a first size smaller than a secondsize of the downsampled color frame. The processor can generate, foreach downsampled color frame of the sequence of downsampled colorframes, a corresponding color intensity value based on the respectiveimage block representing a central image region of the downsampled colorframe. The processor can generate, using color intensity valuescorresponding to the sequence of downsampled color frames, aphotoplethysmographic (PPG) signal of the subject to determine a bloodpressure value of the subject.

In some implementations, the computing device can include a smartphone,tablet, or laptop. In some implementations, the processor can furtherdetermine a blood pressure value of the subject using the PPG signal,and cause display of the blood pressure value of the user on a displaydevice. In determining the blood pressure value of the subject, theprocessor can use an optical blood pressure monitoring (OBPM) algorithm.In some implementations, the respective color frame can be a green colorframe or a red color frame. In some implementations, each downsampledcolor frame can have an n by n size and the respective image blockrepresenting the central image region of the downsampled color frame canhave an m by m size where n and m are integers and m is smaller than n.The integer n can be equal to 5 and m can be equal to 3.

In some implementations, the PPG signal can be a color intensity signalrepresenting average color intensity values of image blocks of thesequence of downsampled color frames. The processor can further filterthe PPG signal using a high-pass filter, and determine the bloodpressure measurement of the subject using a high-pass filtered versionof the PPG signal. The high-pass filter can have a cut-off frequencybetween 0.4 Hz and 4 Hz. The processor can further determine ameasurement of a pulse of the subject using the PPG signal.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause aphotodetector to acquire a sequence of images representing transdermaloptical data of a subject. The one or more processors can generate asequence of downsampled color frames corresponding to the sequence ofimages by downsampling a respective color frame for each image of thesequence of images. The one or more processors can identify, in eachdownsampled color frame of the sequence of downsampled color frames, arespective image block representing a central image region of thedownsampled color frame and having a first size smaller than a secondsize of the downsampled color frame. The one or more processors cangenerate, for each downsampled color frame of the sequence ofdownsampled color frames, a corresponding color intensity value based onthe respective image block representing a central image region of thedownsampled color frame. The one or more processors can generate, usingcolor intensity values corresponding to the sequence of downsampledcolor frames, a photoplethysmographic (PPG) signal of the subject todetermine a blood pressure value of the subject.

According to one aspect, a method can include (a) configuring, accordingto one or more predefined data acquisition requirements, one or moreoperational settings of a photodetector of a computing device, and (b)adjusting, by the computing device, a gain of the photodetector. Themethod can include (c) acquiring, by the photodetector after updatingthe gain, a sequence of images, and (d) generating, using the sequenceof images, a photoplethysmographic (PPG) signal, by determining for eachimage of the sequence of images a corresponding signal value. The methodcan include (e) determining a sequence of local quality values of thePPG signal, where each local quality value is indicative of a localquality of a respective portion of a plurality of portions of the PPGsignal. The method can include (f) determining, by the computing device,a calibration of the photodetector to be successful upon detecting afirst predefined number of consecutive local quality values, among thesequence of local quality values, exceeding a threshold quality.

In some implementations, the method can further include repeating steps(b)-(e) upon detecting a second predefined number of local qualityvalues, among the sequence of local quality values, below the thresholdquality without detecting the first predefined number of consecutivelocal quality values exceeding the threshold quality. In someimplementations, the method can further include repeating steps (b)-(e)for a number of iterations until detecting the first predefined numberof consecutive local quality values exceeding a threshold quality, oruntil the number of iterations reaches a predefined number ofiterations. In some implementations, the method can include providing,by the computing device, an alert message indicative of calibrationtimeout upon the number of iterations reaches the predefined number ofiterations without detecting the first predefined number of consecutivelocal quality values exceeding the threshold quality.

Determining the sequence of local quality values can include determininga sequence of autocorrelation values. Each autocorrelation value of thesequence of autocorrelation values can be associated with a respectivesignal value of the PPG signal. Configuring the one or more operationalsettings of the photodetector can include disabling a color correctionsetting of the photodetector. Configuring the one or more operationalsettings of the photodetector can include disabling an autocorrectionsetting of the photodetector. Configuring the one or more operationalsettings of the photodetector can include enabling a light sourceassociated with the photodetector. Configuring the one or moreoperational settings of the photodetector can include setting a framerate of the photodetector to at least 25 frames per second.

According to one other aspect, a computing device can include aphotodetector, a processor and a memory storing computer codeinstructions. The computer code instructions when executed by theprocessor can cause the processor to (a) configure, according to one ormore predefined data acquisition requirements, one or more operationalsettings of a photodetector of a computing device, and (b) adjust a gainof the photodetector. The processor can (c) cause the photodetector toacquire, after updating the gain, a sequence of images, and (d)generate, using the sequence of images, a photoplethysmographic (PPG)signal, by determining for each image of the sequence of images acorresponding signal value. The processor can (e) determine a sequenceof local quality values of the PPG signal, where each local qualityvalue is indicative of a local quality of a respective portion of aplurality of portions of the PPG signal. The processor can (f) determinea calibration of the photodetector to be successful upon detecting afirst predefined number of consecutive local quality values, among thesequence of local quality values, exceeding a threshold quality.

In some implementations, the processor can further repeat steps (b)-(e)upon detecting a second predefined number of local quality values, amongthe sequence of local quality values, below the threshold qualitywithout detecting the first predefined number of consecutive localquality values exceeding the threshold quality. In some implementations,the processor can repeat steps (b)-(e) for a number of iterations untildetecting the first predefined number of consecutive local qualityvalues exceeding a threshold quality, or until the number of iterationsreaches a predefined number of iterations. In some implementations, theprocessor can provide, by the computing device, an alert messageindicative of calibration timeout upon the number of iterations reachesthe predefined number of iterations without detecting the firstpredefined number of consecutive local quality values exceeding thethreshold quality.

In determining the sequence of local quality values, the processor candetermine a sequence of autocorrelation values. Each autocorrelationvalue of the sequence of autocorrelation values can be associated with arespective signal value of the PPG signal. In configuring the one ormore operational settings of the photodetector, the processor candisable a color correction setting of the photodetector. In configuringthe one or more operational settings of the photodetector, the processorcan disable an autocorrection setting of the photodetector. Inconfiguring the one or more operational settings of the photodetector,the processor can enable a light source associated with thephotodetector. In configuring the one or more operational settings ofthe photodetector, the processor can set a frame rate of thephotodetector to at least 25 frames per second.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to (a) configure, according to one or more predefineddata acquisition requirements, one or more operational settings of aphotodetector of a computing device, and (b) adjust a gain of thephotodetector. The one or more processors can (c) cause thephotodetector to acquire, after updating the gain, a sequence of images,and (d) generate, using the sequence of images, a photoplethysmographic(PPG) signal, by determining for each image of the sequence of images acorresponding signal value. The one or more processors can (e) determinea sequence of local quality values of the PPG signal, where each localquality value is indicative of a local quality of a respective portionof a plurality of portions of the PPG signal. The one or more processorscan (f) determine a calibration of the photodetector to be successfulupon detecting a first predefined number of consecutive local qualityvalues, among the sequence of local quality values, exceeding athreshold quality.

According to one aspect, a method can include acquiring, by aphotodetector of a computing device, a sequence of images, andgenerating, by the computing device, a sequence of downsampled images bydownsampling each image of the sequence of images. The method caninclude, for each downsampled image of the sequence of downsampledimages, the computing device (i) determining, for each pixel of aplurality of pixels in the downsampled image, a respective localvariation value based on a color value of the pixel and one or morecolor values of adjacent pixels, (ii) determining a metric using therespective local variation value of each pixel of the plurality ofpixels of the downsampled image, and (iii) classifying, using themetric, the downsampled image as usable or unusable for measuring bloodpressure.

In some implementations, determining the respective local variationvalue can include determining, by the computing device, a Laplacianvalue for each pixel of the plurality of pixels. Determining the metriccan include determining an average Laplacian value using the respectiveLaplacian value for each pixel. The method may further includenormalizing, by the computing device, the metric by an average colorintensity of the downsampled image. In some implementations, classifyingthe downsampled image can include comparing the metric of thedownsampled image to a threshold. The threshold can be determined usinga machine learning model trained using a plurality of image frames. Themethod can further include generating an alert responsive to determiningthat a predetermined number of downsampled images are classified asunusable. The method can further include computing a blood pressurevalue using a plurality of downsampled images classified as usable.

According to one other aspect, a computing device can include aphotodetector, a processor and a memory storing computer codeinstructions. The computer code instructions when executed by theprocessor can cause the photodetector to acquire a sequence of images,and cause the processor to generate a sequence of downsampled images bydownsampling each image of the sequence of images. The processor can,for each downsampled image of the sequence of downsampled images, (i)determine, for each pixel of a plurality of pixels in the downsampledimage, a respective local variation value based on a color value of thepixel and one or more color values of adjacent pixels, (ii) determine ametric using the respective local variation value of each pixel of theplurality of pixels of the downsampled image, and (iii) classify, usingthe metric, the downsampled image as usable or unusable for measuringblood pressure.

In some implementations, in determining the respective local variationvalue, the processor can determine a Laplacian value for each pixel ofthe plurality of pixels. In determining the metric, the processor candetermine an average Laplacian value using the respective Laplacianvalue for each pixel. The processor may further normalize the metric byan average color intensity of the downsampled image. In someimplementations, in classifying the downsampled image, the processor cancompare the metric of the downsampled image to a threshold. Thethreshold can be determined using a machine learning model trained usinga plurality of image frames. The processor may further generate an alertresponsive to determining that a predetermined number of downsampledimages are classified as unusable. The processor may further compute ablood pressure value using a plurality of downsampled images classifiedas usable.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause thephotodetector to acquire a sequence of images, and cause the one or moreprocessors to generate a sequence of downsampled images by downsamplingeach image of the sequence of images. The one or more processors can,for each downsampled image of the sequence of downsampled images, (i)determine, for each pixel of a plurality of pixels in the downsampledimage, a respective local variation value based on a color value of thepixel and one or more color values of adjacent pixels, (ii) determine ametric using the respective local variation value of each pixel of theplurality of pixels of the downsampled image, and (iii) classify, usingthe metric, the downsampled image as usable or unusable for measuringblood pressure.

According to one aspect, a method can include acquiring, by aphotodetector of a computing device, a sequence of images, andgenerating, by the computing device, a sequence of downsampled images bydownsampling each image of the sequence of images. The method caninclude determining, by the computing device, for each pixel position ofa plurality of pixel positions of the sequence of downsampled images, arespective aggregate pixel similarity score indicative of a similarityof a pixel of the pixel position to pixels of other pixel positions overa time window. The method can include determining, by the computingdevice, using aggregate pixel similarity scores of the plurality ofpixel positions of the sequence of downsampled images, a vectorindicative of a position of a body part of a user.

In some implementations, the method can include determining, for eachpixel position of the plurality of pixel positions of the sequence ofdownsampled images, a respective sequence of pixel intensity valuesrepresenting pixel intensities of the pixel position over the timewindow, and determining, for each pair of pixel positions, a crosscorrelation value representing a cross correlation between a pair ofsequences of pixel intensity values representing pixel intensities ofthe pair of pixel positions over the time window. The method can includedetermining, for each pixel position, the respective aggregate pixelsimilarity score using cross correlation values associated with thepixel position. Determining the respective aggregate pixel similarityscore using cross correlation values associated with the pixel positioncan include (i) determining, for each pair of pixel positions, arespective distance score using the cross correlation value representingthe cross correlation between the pair of sequences of pixel intensityvalues representing pixel intensities of the pair of pixel positionsover the time window, and (ii) determining, for each pixel position, therespective aggregate pixel similarity score as an aggregation ofrespective distance scores associated with the pixel position.

In some implementations, determining the vector can include determininga center of mass of the aggregate pixel similarity scores of theplurality of pixel positions of the downsampled images. The method mayfurther include determining an angle of the vector indicative of adirection along which to move the body part of the user. The method mayfurther include determining a magnitude of the vector indicative of adistance along which to move the body part of the user. The method mayfurther include providing a visual output generated using the vector toguide the user to move the body part relative to the photodetector. Thebody part can be a finger.

According to one other aspect, a computing device can include aphotodetector, a processor and a memory storing computer codeinstructions. The computer code instructions when executed by theprocessor can cause the photodetector to acquire a sequence of images,and cause the processor to generate a sequence of downsampled images bydownsampling each image of the sequence of images. The processor candetermine, for each pixel position of a plurality of pixel positions ofthe sequence of downsampled images, a respective aggregate pixelsimilarity score indicative of a similarity of a pixel of the pixelposition to pixels of other pixel positions over a time window. Theprocessor can determine, using aggregate pixel similarity scores of theplurality of pixel positions of the sequence of downsampled images, avector indicative of a position of a body part of a user.

In some implementations, the processor can determine, for each pixelposition of the plurality of pixel positions of the sequence ofdownsampled images, a respective sequence of pixel intensity valuesrepresenting pixel intensities of the pixel position over the timewindow, and determine, for each pair of pixel positions, a crosscorrelation value representing a cross correlation between a pair ofsequences of pixel intensity values representing pixel intensities ofthe pair of pixel positions over the time window. The processor candetermine, for each pixel position, the respective aggregate pixelsimilarity score using cross correlation values associated with thepixel position. In determining the respective aggregate pixel similarityscore using cross correlation values associated with the pixel position,the processor can (i) determine, for each pair of pixel positions, arespective distance score using the cross correlation value representingthe cross correlation between the pair of sequences of pixel intensityvalues representing pixel intensities of the pair of pixel positionsover the time window, and (ii) determine, for each pixel position, therespective aggregate pixel similarity score as an aggregation ofrespective distance scores associated with the pixel position.

In some implementations, in determining the vector the processor candetermine a center of mass of the aggregate pixel similarity scores ofthe plurality of pixel positions of the downsampled images. Theprocessor may further determine an angle of the vector indicative of adirection along which to move the body part of the user. The processormay further determine a magnitude of the vector indicative of a distancealong which to move the body part of the user. The processor may furtherprovide a visual output generated using the vector to guide the user tomove the body part relative to the photodetector. The body part can be afinger.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause thephotodetector to acquire a sequence of images, and cause the one or moreprocessors to generate a sequence of downsampled images by downsamplingeach image of the sequence of images. The one or more processors candetermine, for each pixel position of a plurality of pixel positions ofthe sequence of downsampled images, a respective aggregate pixelsimilarity score indicative of a similarity of a pixel of the pixelposition to pixels of other pixel positions over a time window. The oneor more processors can determine, using aggregate pixel similarityscores of the plurality of pixel positions of the sequence ofdownsampled images, a vector indicative of a position of a body part ofa user.

According to one aspect, a method can include obtaining, by a computingdevice, a photoplethysmographic (PPG) signal generated from a sequenceof images acquired using a photodetector, and detecting, by thecomputing device, a plurality of signal features of different signalfeature types of the PPG signal. The method can include determining, bythe computing device, using the detected plurality of signal features, apredefined sequence of signal feature types included in a portion of thePPG signal. The predefined sequence of signal feature types can be usedto define PPG pulse segments. The method can include classifying, by thecomputing device, the portion of the PPG signal as a PPG pulse segment.

In some implementations, the plurality of signal feature types caninclude a peak, a trough, a zero-crossing up and a zero-crossing down.The predefined sequence of signal feature types can include an orderedsequence of a first peak, a zero-crossing down, a trough, azero-crossing up and a second peak. The portion of the PPG signal canstart at the first peak and can end at the second peak and the methodcan further include determining, by the computing device, a duration ofthe PPG pulse segment based on a length of time between the first peakand the second peak.

In some implementations, the predefined sequence of signal feature typescan include an ordered sequence of a first trough, a zero-crossing up, apeak, a zero-crossing down and a second trough. The portion of the PPGsignal can start at the first trough and can end at the second trough,and the method can further include determining, by the computing device,a duration of the PPG pulse segment based on a length of time betweenthe first trough and the second trough. In some implementations, theportion of the PPG signal can be a first portion and can end at a firstsignal feature. The method can further include determining, using thedetected plurality of signal features, the predefined sequence of signalfeature types in a second portion of the PPG signal, and classifying thesecond portion of the PPG signal as a second PPG pulse segment. Thesecond portion of the PPG signal can start at the first signal feature.

The method can further include generating, from the PPG signal, aplurality of PPG pulse segments. Each PPG segment can include signalfeatures that include the predefined sequence of signal feature types.The method can include the computing device determining a timedifference between a first signal feature of a first signal feature typeand a second signal feature of a second signal feature type within thePPG pulse segment, comparing the time difference between the firstsignal feature and the second signal feature to a time durationthreshold, and setting a flag to the PPG pulse segment responsive todetermining that time difference between the first signal feature andthe second signal feature exceeds the time duration threshold.

In some implementations, the method can further include the computingdevice (i) generating a sequence of downsampled color framescorresponding to the sequence of images by downsampling a respectivecolor frame for each image of the sequence of images, (ii) identifyingin each downsampled color frame of the sequence of downsampled colorframes, a respective image block representing a central image region ofthe downsampled color frame and having a first size smaller than asecond size of the downsampled color frame, and (iii) generating the PPGsignal using the respective image blocks of the sequence of downsampledcolor frames. The PPG signal can be for use to determine a bloodpressure value. In some implementations, the method can further includethe computing device determining a duration of the PPG pulse segmentbased on a length of time between a first signal feature and a lastsignal feature of the predefined sequence, and determining a type of thePPG pulse segment based on the duration of the PPG pulse segment.

According to one other aspect, a computing device can include aprocessor and a memory storing computer code instructions. The computercode instructions when executed can cause the processor to obtain aphotoplethysmographic (PPG) signal generated from a sequence of imagesacquired using a photodetector, and detect a plurality of signalfeatures of different signal feature types of the PPG signal. Theprocessor can determine, using the detected plurality of signalfeatures, a predefined sequence of signal feature types included in aportion of the PPG signal. The predefined sequence of signal featuretypes can be used to define PPG pulse segments. The processor canclassify the portion of the PPG signal as a PPG pulse segment.

In some implementations, the plurality of signal feature types caninclude a peak, a trough, a zero-crossing up and a zero-crossing down.The predefined sequence of signal feature types can include an orderedsequence of a first peak, a zero-crossing down, a trough, azero-crossing up and a second peak. The portion of the PPG signal canstart at the first peak and can end at the second peak, and theprocessor can further determine a duration of the PPG pulse segmentbased on a length of time between the first peak and the second peak.

In some implementations, the predefined sequence of signal feature typescan include an ordered sequence of a first trough, a zero-crossing up, apeak, a zero-crossing down and a second trough. The portion of the PPGsignal can start at the first trough and can end at the second trough,and the processor can further determine a duration of the PPG pulsesegment based on a length of time between the first trough and thesecond trough. In some implementations, the portion of the PPG signalcan be a first portion and can end at a first signal feature. Theprocessor can further determine, using the detected plurality of signalfeatures, the predefined sequence of signal feature types in a secondportion of the PPG signal, and classify the second portion of the PPGsignal as a second PPG pulse segment. The second portion of the PPGsignal can start at the first signal feature.

The processor can further generate, from the PPG signal, a plurality ofPPG pulse segments. Each PPG segment can include signal features thatinclude the predefined sequence of signal feature types. The processorcan (i) determine a time difference between a first signal feature of afirst signal feature type and a second signal feature of a second signalfeature type within the PPG pulse segment, (ii) compare the timedifference between the first signal feature and the second signalfeature to a time duration threshold, and (iii) set a flag to the PPGpulse segment responsive to determining that time difference between thefirst signal feature and the second signal feature exceeds the timeduration threshold.

In some implementations, the processor can (i) generate a sequence ofdownsampled color frames corresponding to the sequence of images bydownsampling a respective color frame for each image of the sequence ofimages, (ii) identify in each downsampled color frame of the sequence ofdownsampled color frames, a respective image block representing acentral image region of the downsampled color frame and having a firstsize smaller than a second size of the downsampled color frame, and(iii) generate the PPG signal using the respective image blocks of thesequence of downsampled color frames. The PPG signal can be for use todetermine a blood pressure value. In some implementations, the processorcan further determine a duration of the PPG pulse segment based on alength of time between a first signal feature and a last signal featureof the predefined sequence, and determine a type of the PPG pulsesegment based on the duration of the PPG pulse segment.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to obtain a photoplethysmographic (PPG) signalgenerated from a sequence of images acquired using a photodetector, anddetect a plurality of signal features of different signal feature typesof the PPG signal. The one or more processors can determine, using thedetected plurality of signal features, a predefined sequence of signalfeature types included in a portion of the PPG signal. The predefinedsequence of signal feature types can be used to define PPG pulsesegments. The one or more processors can classify the portion of the PPGsignal as a PPG pulse segment.

According to one aspect, a method can include identifying, by acomputing device, from a photoplethysmographic (PPG) signal generatedfrom a sequence of images acquired using a photodetector, a plurality ofPPG pulse segments. Each PPG pulse segment of the plurality of PPG pulsesegments can be identified using a predefined sequence of signal featuretypes. The method can include determining, by the computing device, foreach PPG pulse segment, one or more respective pulse features determinedfrom the PPG pulse segment, and detecting, by the computing device, oneor more conditions based on the one or more respective pulse features ofthe plurality of PPG pulse segments.

In some implementations, the one or more respective pulse features foreach PPG pulse segment can include the respective pulse duration foreach PPG pulse segment. The one or more conditions can include anarrhythmia condition. The method can include determining, for each pairof consecutive PPG pulse segments of a plurality of pairs of consecutivePPG pulse segments, a respective duration ratio of the durationsrepresenting a quantitative relation between durations of the pair ofconsecutive PPG pulse segments, and detecting the arrhythmia conditionusing respective duration ratios for the plurality of pairs ofconsecutive PPG pulse segments. Detecting the arrhythmia condition usingrespective duration ratios for the plurality of pairs of consecutive PPGpulse segments can include (i) determining, by the computing device, areference duration ratio using the respective duration ratios for theplurality of pairs of consecutive PPG pulse segments, (ii) comparing, bythe computing device, the reference duration ratio to a threshold value,and (iii) detecting, by the computing device, the arrhythmia conditionbased on the comparison of the reference duration ratio to the thresholdvalue.

In some implementations, the reference duration ratio can include atleast one of a maximum of the respective duration ratios for theplurality of pairs of consecutive pulse segments, or a quantile of therespective duration ratios for the plurality of pairs of consecutivepulse segments.

In some implementations, the one or more conditions can include anirregular heartbeat condition. The method can include determining, foreach PPG pulse segments of the plurality of PPG pulse segments, arespective pulse duration, determining a first pulse duration of a firstPPG pulse segment and a second pulse duration of a second PPG pulsesegment, and detecting the irregular heartbeat condition based on arelative variation between the first and second pulse durations. Thefirst pulse duration can be a minimum pulse duration of respective pulsedurations of the plurality of PPG pulse segments, and the second pulseduration can be a maximum pulse duration of respective pulse durationsof the plurality of PPG pulse segments. The first pulse duration can bea first quantile of respective pulse durations of the plurality of PPGpulse segments and the second pulse duration can be a second quantile ofrespective pulse durations of the plurality of PPG pulse segments. Therelative variation between the first and second pulse durations caninclude at least one of a ratio of the second pulse duration over thefirst pulse duration, a difference between the first pulse duration andthe second pulse duration or a normalized difference between the firstpulse duration and the second pulse duration.

According to one other aspect, a computing device can include aprocessor and a memory storing computer code instructions. The computercode instructions when executed can cause the processor to identify,from a photoplethysmographic (PPG) signal generated from a sequence ofimages acquired using a photodetector, a plurality of PPG pulsesegments. Each PPG pulse segment of the plurality of PPG pulse segmentscan be identified using a predefined sequence of signal feature types.The processor can determine, for each PPG pulse segment, one or morerespective pulse features determined from the PPG pulse segment, anddetect one or more conditions based on the one or more respective pulsefeatures of the plurality of PPG pulse segments.

In some implementations, the one or more respective pulse features foreach PPG pulse segment can include the respective pulse duration foreach PPG pulse segment. The one or more conditions can include anarrhythmia condition. The processor can determine, for each pair ofconsecutive PPG pulse segments of a plurality of pairs of consecutivePPG pulse segments, a respective duration ratio of the durationsrepresenting a quantitative relation between durations of the pair ofconsecutive PPG pulse segments, and detect the arrhythmia conditionusing respective duration ratios for the plurality of pairs ofconsecutive PPG pulse segments. In detecting the arrhythmia conditionusing respective duration ratios for the plurality of pairs ofconsecutive PPG pulse segments, the processor can (i) determine areference duration ratio using the respective duration ratios for theplurality of pairs of consecutive PPG pulse segments, (ii) compare thereference duration ratio to a threshold value, and (iii) detect thearrhythmia condition based on the comparison of the reference durationratio to the threshold value.

In some implementations, the reference duration ratio can include atleast one of a maximum of the respective duration ratios for theplurality of pairs of consecutive pulse segments, or a quantile of therespective duration ratios for the plurality of pairs of consecutivepulse segments.

In some implementations, the one or more conditions can include anirregular heartbeat condition. The processor can determine, for each PPGpulse segments of the plurality of PPG pulse segments, a respectivepulse duration, determining a first pulse duration of a first PPG pulsesegment and a second pulse duration of a second PPG pulse segment, anddetect the irregular heartbeat condition based on a relative variationbetween the first and second pulse durations. The first pulse durationcan be a minimum pulse duration of respective pulse durations of theplurality of PPG pulse segments, and the second pulse duration can be amaximum pulse duration of respective pulse durations of the plurality ofPPG pulse segments. The first pulse duration can be a first quantile ofrespective pulse durations of the plurality of PPG pulse segments andthe second pulse duration can be a second quantile of respective pulsedurations of the plurality of PPG pulse segments. The relative variationbetween the first and second pulse durations can include at least one ofa ratio of the second pulse duration over the first pulse duration, adifference between the first pulse duration and the second pulseduration or a normalized difference between the first pulse duration andthe second pulse duration.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to identify, from a photoplethysmographic (PPG)signal generated from a sequence of images acquired using aphotodetector, a plurality of PPG pulse segments. Each PPG pulse segmentof the plurality of PPG pulse segments can be identified using apredefined sequence of signal feature types. The one or more processorscan determine, for each PPG pulse segment, one or more respective pulsefeatures determined from the PPG pulse segment, and detect one or moreconditions based on the one or more respective pulse features of theplurality of PPG pulse segments.

According to one aspect, a method can include obtaining, by a computingdevice, a photoplethysmographic (PPG) signal generated from a sequenceof images acquired using a photodetector while a body part is placed inproximity of the photodetector. The method can include determining, bythe computing device, a logarithmic PPG signal by computing a logarithmof the PPG signal. The logarithmic PPG signal can be indicative of bloodabsorbance of light. The method can include the computing devicedetermining an envelope of the logarithmic PPG signal, and determining ablood perfusion condition of the body part based on the envelope of thelogarithmic PPG signal.

In some implementations, the blood perfusion condition can include a lowblood perfusion condition indicative of a cold body part. The method canfurther include providing an alert indicative of the cold body part. Thealert can include one or more instructions for warming up the body part.Determining the envelope of the logarithmic PPG signal can includeapplying a low-pass filter to the logarithmic PPG signal.

In some implementations, determining the envelope of the logarithmic PPGsignal can include determining for each time interval of a plurality oftime intervals a respective maximum and a respective minimum of thelogarithmic PPG signal. The method can further include determining, foreach time interval of the plurality of time intervals, a respectivelocal variation of the envelope of the logarithmic PPG signal. Therespective local variation can be equal to a difference between therespective maximum and the respective minimum of the logarithmic PPGsignal within the time interval. The method can determine an estimate ofan amplitude of the logarithmic PPG signal based using respective localvariations of the envelope of the logarithmic PPG signal within theplurality of time intervals. The blood perfusion condition of the bodypart can be determined based on the estimate of the amplitude of thelogarithmic PPG signal.

Determining the blood perfusion condition of the body part can includethe computing device comparing the estimate of the amplitude of thelogarithmic PPG signal to a threshold value, and determining the bloodperfusion condition of the body part based on the comparison of theestimate of the amplitude of the logarithmic PPG signal to the thresholdvalue. Determining the estimate of the amplitude of the logarithmic PPGsignal can include at least one of (i) determining the estimate of theamplitude of the logarithmic PPG signal as a predefined quantile of therespective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals, (ii) determining theestimate of the amplitude of the logarithmic PPG signal as a median ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals (iii) determining theestimate of the amplitude of the logarithmic PPG signal as an average ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals (iv) determining theestimate of the amplitude of the logarithmic PPG signal as a maximum ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals, or (v) determining theestimate of the amplitude of the logarithmic PPG signal as a minimum ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals.

In some implementations, the method can include the computing devicegenerating a sequence of downsampled color frames corresponding to thesequence of images by downsampling a respective color frame for eachimage of the sequence of images, identifying, in each downsampled colorframe of the sequence of downsampled color frames, a respective imageblock representing a central image region of the downsampled color frameand having a first size smaller than a second size of the downsampledcolor frame, and generating the PPG signal using the respective imageblocks of the sequence of downsampled color frames. The PPG signal canbe used to determine a blood pressure value.

According to one other aspect, a computing device can include aprocessor and a memory storing computer code instructions. The computercode instructions when executed can cause the processor to obtain aphotoplethysmographic (PPG) signal generated from a sequence of imagesacquired using a photodetector while a body part is placed in proximityof the photodetector. The processor can determine a logarithmic PPGsignal by computing a logarithm of the PPG signal. The logarithmic PPGsignal can be indicative of blood absorbance of light. The processor candetermine an envelope of the logarithmic PPG signal, and determine ablood perfusion condition of the body part based on the envelope of thelogarithmic PPG signal.

In some implementations, the blood perfusion condition can include a lowblood perfusion condition indicative of a cold body part. The processorcan further provide an alert indicative of the cold body part. The alertcan include one or more instructions for warming up the body part. Indetermining the envelope of the logarithmic PPG signal, the processorcan apply a low-pass filter to the logarithmic PPG signal.

In some implementations, in determining the envelope of the logarithmicPPG signal, the processor can determine for each time interval of aplurality of time intervals a respective maximum and a respectiveminimum of the logarithmic PPG signal. The processor can furtherdetermine, for each time interval of the plurality of time intervals, arespective local variation of the envelope of the logarithmic PPGsignal. The respective local variation can be equal to a differencebetween the respective maximum and the respective minimum of thelogarithmic PPG signal within the time interval. The processor candetermine an estimate of an amplitude of the logarithmic PPG signalusing respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals. The blood perfusioncondition of the body part can be determined based on the estimate ofthe amplitude of the logarithmic PPG signal.

In determining the blood perfusion condition of the body part, theprocessor can compare the estimate of the amplitude of the logarithmicPPG signal to a threshold value, and determine the blood perfusioncondition of the body part based on the comparison of the estimate ofthe amplitude of the logarithmic PPG signal to the threshold value. Indetermining the estimate of the amplitude of the logarithmic PPG signal,the processor can perform at least one of (i) determine the estimate ofthe amplitude of the logarithmic PPG signal as a predefined quantile ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals, (ii) determine theestimate of the amplitude of the logarithmic PPG signal as a median ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals (iii) determine theestimate of the amplitude of the logarithmic PPG signal as an average ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals (iv) determine theestimate of the amplitude of the logarithmic PPG signal as a maximum ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals, or (v) determine theestimate of the amplitude of the logarithmic PPG signal as a minimum ofthe respective local variations of the envelope of the logarithmic PPGsignal within the plurality of time intervals.

In some implementations, the processor can generate a sequence ofdownsampled color frames corresponding to the sequence of images bydownsampling a respective color frame for each image of the sequence ofimages. The processor can identify, in each downsampled color frame ofthe sequence of downsampled color frames, a respective image blockrepresenting a central image region of the downsampled color frame andhaving a first size smaller than a second size of the downsampled colorframe. The processor can generate the PPG signal using the respectiveimage blocks of the sequence of downsampled color frames. The PPG signalcan be used to determine a blood pressure value.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to obtain a photoplethysmographic (PPG) signalgenerated from a sequence of images acquired using a photodetector whilea body part is placed in proximity of the photodetector. The one or moreprocessors can determine a logarithmic PPG signal by computing alogarithm of the PPG signal. The logarithmic PPG signal can beindicative of blood absorbance of light. The one or more processors candetermine an envelope of the logarithmic PPG signal, and determine ablood perfusion condition of the body part based on the envelope of thelogarithmic PPG signal.

According to one aspect, a method can include obtaining, by a computingdevice, a sequence of images acquired by a photodetector. The method caninclude determining, by the computing device, for each pixel position ofa plurality of pixel positions associated with the sequence of images, arespective penalty score indicative of a similarity between a colorvalue of a pixel of the pixel position and a desired color value. Thedesired color value can represent a color property of light emitted frombody parts of users when placed opposite to the photodetector. Themethod can include determining, by the computing device, using penaltyscores of the plurality of pixel positions, a relative position of abody part of a user with respect to a desired position.

In some implementations, the method can include generating, by thecomputing device, a sequence of downsampled images by downsampling eachimage of the sequence of images. The plurality of pixel positions canrepresent pixel positions of the sequence of downsampled images. In someimplementations, the method can include transforming, by the computingdevice, the sequence of images to a hue, saturation, luminance (HSL)color space. The color value can represent a hue color value of thepixel of the pixel position and the desired color value representing adesired hue color value. The penalty score can be defined as a functionof min(|H−T_(H)|, 360−|H−T_(H)|), where H represents the hue color valueand T_(H) represents the desired hue color value. In someimplementations, the penalty score can be defined in terms of adifference between the color value and the desired color value.

In some implementations, determining the relative position can includedetermining a center of mass of the penalty scores of the plurality ofpixel positions. Determining the relative position can includedetermining a magnitude and a direction of a position vector. Theposition vector can represent the relative position with respect to thedesired position. The method can include providing a visual outputindicative of the relative position for display on a display device. Thevisual output can include an indication of a classification of therelative position of the body part. The body part can be a finger of theuser.

According to one other aspect, a computing device can include aprocessor and a memory storing computer code instructions. The computercode instructions when executed can cause the processor to obtain asequence of images acquired by a photodetector. The processor candetermine, for each pixel position of a plurality of pixel positionsassociated with the sequence of images, a respective penalty scoreindicative of a similarity between a color value of a pixel of the pixelposition and a desired color value. The desired color value canrepresent a color property of light emitted from body parts of userswhen placed opposite to the photodetector. The processor can determine,using penalty scores of the plurality of pixel positions, a relativeposition of a body part of a user with respect to a desired position.

In some implementations, the processor can generate a sequence ofdownsampled images by downsampling each image of the sequence of images.The plurality of pixel positions can represent pixel positions of thesequence of downsampled images. In some implementations, the processorcan transform the sequence of images to a hue, saturation, luminance(HSL) color space. The color value can represent a hue color value ofthe pixel of the pixel position and the desired color value representinga desired hue color value. The penalty score can be defined as afunction of min(|H−T_(H)|, 360−|H−T_(H)|), where H represents the huecolor value and T_(H) represents the desired hue color value. In someimplementations, the penalty score can be defined in terms of adifference between the color value and the desired color value.

In some implementations, in determining the relative position, theprocessor can determine a center of mass of the penalty scores of theplurality of pixel positions. Determining the relative position caninclude determining a magnitude and a direction of a position vector.The position vector can represent the relative position with respect tothe desired position. The processor can provide a visual outputindicative of the relative position for display on a display device. Thevisual output can include an indication of a classification of therelative position of the body part. The body part can be a finger of theuser.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to obtain a sequence of images acquired by aphotodetector. The one or more processors can determine, for each pixelposition of a plurality of pixel positions associated with the sequenceof images, a respective penalty score indicative of a similarity betweena color value of a pixel of the pixel position and a desired colorvalue. The desired color value can represent a color property of lightemitted from body parts of users when placed opposite to thephotodetector. The one or more processors can determine, using penaltyscores of the plurality of pixel positions, a relative position of abody part of a user with respect to a desired position.

According to one aspect, a method can include obtaining, by a computingdevice, a sequence of images representing transdermal optical data of asubject. The sequence of images can be acquired by a photodetector. Themethod can include identifying, by the computing device, a plurality ofimage regions across the sequence of images. The method can includegenerating, by the computing device, a plurality of color intensitysignals associated with the plurality of image regions across thesequence of images. The method can include determining, by the computingdevice, using a machine learning model and the plurality of colorintensity signals associated with the plurality of image regions, acondition associated with acquisition of the sequence of images. Themachine learning model can receive the plurality color intensity signalsas input and provide an indication of the condition as output. Themethod can include providing, by the computing device, feedback forpresentation to a user based on the condition associated with theacquisition of the sequence of images.

In some implementations, identifying the image regions across thesequence of images can include downsampling, for each image of thesequence of images, corresponding red and green color frames, andidentifying, for each image of the sequence of images, the plurality ofimage regions across downsampled green color image frames and acrossdownsampled red color frames. In some implementations, generating theplurality of color intensity signals can include at least one ofgenerating one or more green color intensity signals associated with oneor more image regions across the downsampled green color frames, orgenerating one or more red color intensity signals associated with oneor more image regions across the downsampled red color frames.

In some implementations, the plurality of image regions can include acentral image region and one or more side image regions. In someimplementations, generating a color intensity signal associated with animage region across the sequence of images can include averaging pixelcolor values of the image region for each image of the sequence ofimages. The condition can include at least one of a condition related toplacement of a body part of the subject relative to the photodetector ora blood perfusion condition of the subject. The condition can include acolor intensity saturation condition of the sequence of images. Themethod can further include training the machine learning model using asecond plurality of color intensity signals from one or more secondimage sequences. The second plurality of color intensity signals can beassociated with the plurality of image regions across the one or moresecond image sequences.

According to one other aspect, a computing device can include aprocessor and a memory storing computer code instructions. The computercode instructions when executed can cause the processor to obtain asequence of images representing transdermal optical data of a subject.The sequence of images can be acquired by a photodetector. The processorcan identify a plurality of image regions across the sequence of images.The processor can generate a plurality of color intensity signalsassociated with the plurality of image regions across the sequence ofimages. The processor can determine, using a machine learning model andthe plurality of color intensity signals associated with the pluralityof image regions, a condition associated with acquisition of thesequence of images. The machine learning model can receive the pluralitycolor intensity signals as input and provide an indication of thecondition as output. The processor can provide feedback for presentationto a user based on the condition associated with the acquisition of thesequence of images.

In some implementations, in identifying the image regions across thesequence of images, the processor can downsample, for each image of thesequence of images, corresponding red and green color frames, andidentify, for each image of the sequence of images, the plurality ofimage regions across downsampled green color image frames and acrossdownsampled red color frames. In some implementations, in generating theplurality of color intensity signals, the processor can generate one ormore green color intensity signals associated with one or more imageregions across the downsampled green color frames, or generate one ormore red color intensity signals associated with one or more imageregions across the downsampled red color frames.

In some implementations, the plurality of image regions can include acentral image region and one or more side image regions. In someimplementations, in generating a color intensity signal associated withan image region across the sequence of images, the processor can averagepixel color values of the image region for each image of the sequence ofimages. The condition can include at least one of a condition related toplacement of a body part of the subject relative to the photodetector ora blood perfusion condition of the subject. The condition can include acolor intensity saturation condition of the sequence of images. Theprocessor can further train the machine learning model using a secondplurality of color intensity signals from one or more second imagesequences. The second plurality of color intensity signals can beassociated with the plurality of image regions across the one or moresecond image sequences.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to obtain a sequence of images representingtransdermal optical data of a subject. The sequence of images can beacquired by a photodetector. The one or more processors can identify aplurality of image regions across the sequence of images. The processorcan generate a plurality of color intensity signals associated with theplurality of image regions across the sequence of images. The one ormore processors can determine, using a machine learning model and theplurality of color intensity signals associated with the plurality ofimage regions, a condition associated with acquisition of the sequenceof images. The machine learning model can receive the plurality colorintensity signals as input and provide an indication of the condition asoutput. The one or more processors can provide feedback for presentationto a user based on the condition associated with the acquisition of thesequence of images.

According to one aspect, a method of assessing PPG signals generatedbased on transdermal optical data can include generating, by a computingdevice, a color intensity signal using an acquired sequence oftransdermal images of a subject. The method can include computing, bythe computing device, a signal quality metric of the color intensitysignal. The method can include providing, by the computing device, anindication of a quality of the color intensity signal for display on adisplay device associated with the computing device. The indication ofthe quality of the color intensity signal can be determined based on thesignal quality metric.

The signal quality metric can include a normalized autocorrelationscore. The signal quality metric can include a spectrum entropy basedsignal quality metric. The spectrum entropy based signal quality metriccan be defined based on a normalized power spectrum of the colorintensity signal. The method can further include comparing, by thecomputing device, the signal quality metric to a corresponding thresholdvalue, and determining, by the computing device, the indication of thequality of the color intensity signal based on comparing the signalquality metric to the corresponding threshold value. Computing thesignal quality metric can include computing a plurality of values of thesignal quality metric based on a sliding window. The method can furtherinclude determining, by the computing device, a local period of thecolor intensity signal using the plurality of values of the signalquality metric. Determining the local period of the color intensitysignal can include determining a maximum of normalized autocorrelationparameters computed based on a sliding window.

The method can further include comparing, by the computing device, eachvalue of the plurality of values of the signal quality metric to acorresponding threshold value, and determining, by the computing device,for each value of the plurality of values of the signal quality metric,a corresponding indication of the quality of the color intensity signalbased on comparing the value of signal quality metric to thecorresponding threshold value. The method can include aborting, by thecomputing device, the recording if the multiple indications of thequality of the color intensity signal corresponding to a predefinedcumulative time period are below the corresponding threshold value. Themethod can include at least one of displaying a graph of at least aportion of the color intensity signal, displaying a signal quality bardepicting a visual representation of the indication of the quality ofthe color intensity signal or displaying a timer indicative of time leftfor acquiring the sequence of transdermal images of the subject.

According to one other aspect, a computing device can include aprocessor and a memory storing computer code instructions. The computercode instructions when executed can cause the processor to generate acolor intensity signal using an acquired sequence of transdermal imagesof a subject, and compute a signal quality metric of the color intensitysignal. The processor can provide an indication of a quality of thecolor intensity signal for display on a display device associated withthe computing device. The indication of the quality of the colorintensity signal can be determined based on the signal quality metric.

The signal quality metric can include a normalized autocorrelationscore. The signal quality metric can include a spectrum entropy basedsignal quality metric. The spectrum entropy based signal quality metriccan be defined based on a normalized power spectrum of the colorintensity signal. The processor can compare the signal quality metric toa corresponding threshold value, and determine the indication of thequality of the color intensity signal based on comparing the signalquality metric to the corresponding threshold value. In computing thesignal quality metric, the processor can compute a plurality of valuesof the signal quality metric based on a sliding window. The processorcan determine a local period of the color intensity signal using theplurality of values of the signal quality metric. In determining thelocal period of the color intensity signal, the processor can determinea maximum of normalized autocorrelation parameters computed based on asliding window.

The processor can further compare each value of the plurality of valuesof the signal quality metric to a corresponding threshold value, anddetermine for each value of the plurality of values of the signalquality metric, a corresponding indication of the quality of the colorintensity signal based on comparing the value of signal quality metricto the corresponding threshold value. The processor can abort therecording if the multiple indications of the quality of the colorintensity signal corresponding to a predefined cumulative time periodare below the corresponding threshold value. The processor can performat least one of display a graph of at least a portion of the colorintensity signal, display a signal quality bar depicting a visualrepresentation of the indication of the quality of the color intensitysignal or display a timer indicative of time left for acquiring thesequence of transdermal images of the subject.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to generate a color intensity signal using anacquired sequence of transdermal images of a subject, and compute asignal quality metric of the color intensity signal. The one or moreprocessors can provide an indication of a quality of the color intensitysignal for display on a display device associated with the computingdevice. The indication of the quality of the color intensity signal canbe determined based on the signal quality metric.

According to one aspect, a method can include associating, by acomputing device, a calibration photoplethysmographic (PPG) signalgenerated from a first sequence of image frames obtained from aphotodetector of the computing device with one or more measurementvalues generated by a blood pressure measurement device different fromthe computing device. The method can include obtaining, by the computingdevice, a recording PPG signal generated from a second sequence of imageframes obtained from the photodetector, and identifying, by thecomputing device, a calibration model from a plurality of blood pressurecalibration models based on the calibration PPG signal and the recordingPPG signal. The method can include generating, by the computing device,a calibrated blood pressure value using the recording PPG signal,features associated with the calibration PPG signal and the identifiedcalibration model.

In some implementations, the method can further include the computingdevice generating, from the recording PPG signal, a logarithmicrecording PPG signal, and generating, from the calibration PPG signal, alogarithmic calibration PPG signal. In some implementations, the methodcan include the computing device determining whether the recording PPGsignal generates a first blood pressure estimate, determining whetherthe logarithmic recording PPG signal generates a second blood pressureestimate, determining whether the calibration PPG signal generates athird blood pressure estimate, and determining whether the logarithmiccalibration PPG signal generates a fourth blood pressure estimate.Identifying the calibration model can include selecting the calibrationmodel from a plurality of calibration models based on (i) whether therecording PPG signal generates the first blood pressure estimate, (ii)whether the logarithmic recording PPG signal generates the second bloodpressure estimate, (iii) whether the calibration PPG signal generatesthe third blood pressure estimate, and (iv) whether the logarithmiccalibration PPG signal generates the fourth blood pressure estimate.

In some implementations, each blood pressure calibration model of theplurality of blood pressure calibration models can be associated with acorresponding set of parameter variables used to determine calibratedblood pressure values. The corresponding set of parameter variables canincludes at least one of (i) one or more parameter variables indicativeof one or more recording signal features extracted from a logarithmicrecording PPG signal where the logarithmic recording PPG signal can begenerated from the recording PPG signal, or (ii) one or more parametervariables indicative of one or more calibration signal featuresextracted from a logarithmic calibration PPG signal where thelogarithmic calibration PPG signal can be generated from the calibrationPPG signal. The corresponding set of parameter variables can include atleast one of (i) one or more parameter variables indicative of one ormore first pulse related features extracted from pulses of the recordingPPG signal, or (ii) one or more parameter variables indicative of one ormore second pulse related features extracted from pulses of thecalibration PPG signal.

In some implementations, the corresponding set of parameter variablescan include one or more parameter variables indicative of one or morecalibration features of the calibration PPG signal. The one or morecalibration features can include at least one of (i) a first systolicblood pressure estimate generated using the calibration PPG signal as aninput signal, (ii) a first diastolic blood pressure estimate generatedusing the calibration PPG signal as an input signal, (iii) a secondsystolic blood pressure estimate generated using a logarithmiccalibration PPG signal as input signal where the logarithmic calibrationPPG signal can be generated from the calibration PPG signal, or (iv) asecond diastolic blood pressure estimate generated using the logarithmiccalibration PPG signal as input signal.

In some implementations, the corresponding set of parameter variablescan include one or more parameter variables indicative of one or morefeatures of the recording PPG signal. The one or more features of therecording PPG signal can include at least one of (i) a first systolicblood pressure estimate generated using the recording PPG signal as aninput signal, (ii) a first diastolic blood pressure estimate generatedusing the recording PPG signal as an input signal, (iii) a secondsystolic blood pressure estimate generated using a logarithmic recordingPPG signal as input signal where the logarithmic recording PPG signalcan be generated from the recording PPG signal, or (iv) a seconddiastolic blood pressure estimate generated using the logarithmicrecording PPG signal as input signal. The corresponding set of parametervariables can include one or more parameter variables indicative of oneor more demographic features of a user of the computing device. Theplurality of blood pressure calibration models can include one or moremachine learning models and the method can further include training eachmachine learning model of the one or more machine learning models usinglabeled data to determine the corresponding set of parameter variables.According to one other aspect, a computing device can include aprocessor and a memory storing computer code instructions. The computercode instructions when executed can cause the processor to associate acalibration photoplethysmographic (PPG) signal generated from a firstsequence of image frames obtained from a photodetector of the computingdevice with one or more measurement values generated by a blood pressuremeasurement device different from the computing device. The processorcan obtain a recording PPG signal generated from a second sequence ofimage frames obtained from the photodetector, and identify a calibrationmodel from a plurality of blood pressure calibration models based on thecalibration PPG signal and the recording PPG signal. The processor cangenerate a calibrated blood pressure value using the recording PPGsignal, features associated with the calibration PPG signal and theidentified calibration model.

In some implementations, the processor can further generate, from therecording PPG signal, a logarithmic recording PPG signal, and generate,from the calibration PPG signal, a logarithmic calibration PPG signal.In some implementations, the processor can determine whether therecording PPG signal generates a first blood pressure estimate,determine whether the logarithmic recording PPG signal generates asecond blood pressure estimate, determine whether the calibration PPGsignal generates a third blood pressure estimate, and determine whetherthe logarithmic calibration PPG signal generates a fourth blood pressureestimate. In identifying the calibration model, the processor can selectthe calibration model from a plurality of calibration models based on(i) whether the recording PPG signal generates the first blood pressureestimate, (ii) whether the logarithmic recording PPG signal generatesthe second blood pressure estimate, (iii) whether the calibration PPGsignal generates the third blood pressure estimate, and (iv) whether thelogarithmic calibration PPG signal generates the fourth blood pressureestimate.

In some implementations, each blood pressure calibration model of theplurality of blood pressure calibration models can be associated with acorresponding set of parameter variables used to determine calibratedblood pressure values. The corresponding set of parameter variables caninclude at least one of (i) one or more parameter variables indicativeof one or more recording signal features extracted from a logarithmicrecording PPG signal where the logarithmic recording PPG signal can begenerated from the recording PPG signal, or (ii) one or more parametervariables indicative of one or more calibration signal featuresextracted from a logarithmic calibration PPG signal where thelogarithmic calibration PPG signal can be generated from the calibrationPPG signal. The corresponding set of parameter variables can include atleast one of (i) one or more parameter variables indicative of one ormore first pulse related features extracted from pulses of the recordingPPG signal, or (ii) one or more parameter variables indicative of one ormore second pulse related features extracted from pulses of thecalibration PPG signal.

In some implementations, the corresponding set of parameter variablescan include one or more parameter variables indicative of one or morecalibration features of the calibration PPG signal. The one or morecalibration features can include at least one of (i) a first systolicblood pressure estimate generated using the calibration PPG signal as aninput signal, (ii) a first diastolic blood pressure estimate generatedusing the calibration PPG signal as an input signal, (iii) a secondsystolic blood pressure estimate generated using a logarithmiccalibration PPG signal as input signal where the logarithmic calibrationPPG signal can be generated from the calibration PPG signal, or (iv) asecond diastolic blood pressure estimate generated using the logarithmiccalibration PPG signal as input signal.

In some implementations, the corresponding set of parameter variablescan include one or more parameter variables indicative of one or morefeatures of the recording PPG signal. The one or more features of therecording PPG signal can include at least one of (i) a first systolicblood pressure estimate generated using the recording PPG signal as aninput signal, (ii) a first diastolic blood pressure estimate generatedusing the recording PPG signal as an input signal, (iii) a secondsystolic blood pressure estimate generated using a logarithmic recordingPPG signal as input signal where the logarithmic recording PPG signalcan be generated from the recording PPG signal, or (iv) a seconddiastolic blood pressure estimate generated using the logarithmicrecording PPG signal as input signal. The corresponding set of parametervariables can include one or more parameter variables indicative of oneor more demographic features of a user of the computing device. Theplurality of blood pressure calibration models can include one or moremachine learning models and the processor can further train each machinelearning model of the one or more machine learning models using labeleddata to determine the corresponding set of parameter variables.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to associate a calibration photoplethysmographic(PPG) signal generated from a first sequence of image frames obtainedfrom a photodetector of a computing device with one or more measurementvalues generated by a blood pressure measurement device different fromthe computing device. The one or more processors can obtain a recordingPPG signal generated from a second sequence of image frames obtainedfrom the photodetector, and identify a calibration model from aplurality of blood pressure calibration models based on the calibrationPPG signal and the recording PPG signal. The one or more processors cangenerate a calibrated blood pressure value using the recording PPGsignal, features associated with the calibration PPG signal and theidentified calibration model.

According to one aspect, a method can include a computing deviceobtaining a photoplethysmographic (PPG) generated from a sequence ofimages obtained from a photodetector, and identifying a blood pressureestimation model from a plurality of machine learning blood pressureestimation models based on the PPG signal. The method can include thecomputing device generating a blood pressure value using the PPG signaland the identified blood pressure estimation model.

The method can further include the computing device generating, from thePPG signal, a logarithmic PPG signal. The method can further includedetermining whether the PPG signal generates a first blood pressureestimate, and determining whether the logarithmic PPG signal generates asecond blood pressure estimate. Identifying the blood pressureestimation model can include selecting the blood pressure estimationmodel from a plurality of blood pressure estimation models based on (i)whether the PPG signal generates the first blood pressure estimate and(ii) whether the logarithmic PPG signal generates the second bloodpressure estimate.

In some implementations, the blood pressure estimation model can beselected from a plurality of machine learning models. Each machinelearning model of the plurality of machine learning models can beassociated with a corresponding set of parameter variables used todetermine calibrated blood pressure values. The corresponding set ofparameter variables can include one or more parameter variablesindicative of one or more signal features extracted from a logarithmicPPG signal, the logarithmic PPG signal generated from the PPG signal.The corresponding set of parameter variables can include one or moreparameter variables indicative of one or more pulse related featuresextracted from pulses of the PPG signal.

In some implementations, the corresponding set of parameter variablescan include one or more parameter variables indicative of one or morefeatures of the PPG signal. The one or more features of the PPG signalcan include at least one of (i) a first systolic blood pressure estimategenerated using the PPG signal as an input signal, (ii) a firstdiastolic blood pressure estimate generated using the PPG signal as aninput signal, (iii) a second systolic blood pressure estimate generatedusing a logarithmic PPG signal as an input signal, the logarithmic PPGsignal generated from the PPG signal, or (iv) a second diastolic bloodpressure estimate generated using the logarithmic PPG signal as an inputsignal. The corresponding set of parameter variables can include one ormore parameter variables indicative of one or more demographic featuresof a user of the computing device. The method can further includetraining each machine learning model of the plurality of machinelearning models using labeled data to determine the corresponding set ofparameter variables. The blood pressure estimation model can include alinear regression model or a nonlinear regression model. According toone other aspect, a computing device can include a processor and amemory storing computer code instructions. The computer codeinstructions when executed can cause the processor to obtain aphotoplethysmographic (PPG) generated from a sequence of images obtainedfrom a photodetector of the computing device, and identify a bloodpressure estimation model from a plurality of machine learning bloodpressure estimation models based on the PPG signal. The processor cangenerate a blood pressure value using the PPG signal and the identifiedblood pressure estimation model.

The processor can generate, from the PPG signal, a logarithmic PPGsignal. The processor can determine whether the PPG signal generates afirst blood pressure estimate, and determine whether the logarithmic PPGsignal generates a second blood pressure estimate. In identifying theblood pressure estimation model, the processor can select the bloodpressure estimation model from a plurality of blood pressure estimationmodels based on (i) whether the PPG signal generates the first bloodpressure estimate and (ii) whether the logarithmic PPG signal generatesthe second blood pressure estimate.

In some implementations, the blood pressure estimation model can beselected from a plurality of machine learning models. Each machinelearning model of the plurality of machine learning models can beassociated with a corresponding set of parameter variables used todetermine calibrated blood pressure values. The corresponding set ofparameter variables can include one or more parameter variablesindicative of one or more signal features extracted from a logarithmicPPG signal, the logarithmic PPG signal generated from the PPG signal.The corresponding set of parameter variables can include one or moreparameter variables indicative of one or more pulse related featuresextracted from pulses of the PPG signal.

In some implementations, the corresponding set of parameter variablescan include one or more parameter variables indicative of one or morefeatures of the PPG signal. The one or more features of the PPG signalcan include at least one of (i) a first systolic blood pressure estimategenerated using the PPG signal as an input signal, (ii) a firstdiastolic blood pressure estimate generated using the PPG signal as aninput signal, (iii) a second systolic blood pressure estimate generatedusing a logarithmic PPG signal as an input signal, the logarithmic PPGsignal generated from the PPG signal, or (iv) a second diastolic bloodpressure estimate generated using the logarithmic PPG signal as an inputsignal. The corresponding set of parameter variables can include one ormore parameter variables indicative of one or more demographic featuresof a user of the computing device. The method can further includetraining each machine learning model of the plurality of machinelearning models using labeled data to determine the corresponding set ofparameter variables. The blood pressure estimation model can include alinear regression model or a nonlinear regression model.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to obtain a photoplethysmographic (PPG) generatedfrom a sequence of images obtained from a photodetector of the computingdevice, and identify a blood pressure estimation model from a pluralityof machine learning blood pressure estimation models based on the PPGsignal. The one or more processors can generate a blood pressure valueusing the PPG signal and the identified blood pressure estimation model.

According to one aspect, a method can include obtaining, by a computingdevice, one or more photoplethysmographic (PPG) signals generated from asequence of images obtained from a photodetector of the computingdevice. The method can include extracting, by the computing device, aplurality of features of the PPG signal. The method can includegenerating, by the computing device, a blood pressure classificationusing the plurality of features of the PPG signal and a blood pressureclassification model.

In some implementations, the blood pressure classification model can bea machine learning model associated with a corresponding set ofparameter variables used to determine the blood pressure classification.The method can further include training the machine learning model usinglabeled data to determine the corresponding set of parameter variables.The plurality of features of the PPG signal can include one or moresignal features extracted from a logarithmic PPG signal generated fromthe PPG signal. The plurality of features of the PPG signal can includeone or more pulse related features extracted from pulses of the PPGsignal.

The plurality of features of the PPG signal can include at least one of(i) a first systolic blood pressure estimate generated using the PPGsignal as an input signal, (ii) a first diastolic blood pressureestimate generated using the PPG signal as an input signal, (iii) asecond systolic blood pressure estimate generated using a logarithmicPPG signal as an input signal where the logarithmic PPG signal can begenerated from the PPG signal, or (iv) a second diastolic blood pressureestimate generated using the logarithmic PPG signal as an input signal.The blood pressure classification can be generated using one or moredemographic features of a user of the computing device.

According to one other aspect, a computing device can include aprocessor and a memory storing computer code instructions. The computercode instructions when executed can cause the processor to obtain one ormore photoplethysmographic (PPG) signals generated from a sequence ofimages obtained from a photodetector of the computing device. Theprocessor can extract a plurality of features of the PPG signal. Theprocessor can generate a blood pressure classification using theplurality of features of the PPG signal and a blood pressureclassification model.

In some implementations, the blood pressure classification model can bea machine learning model associated with a corresponding set ofparameter variables used to determine the blood pressure classification.The processor can further train the machine learning model using labeleddata to determine the corresponding set of parameter variables. Theplurality of features of the PPG signal can include one or more signalfeatures extracted from a logarithmic PPG signal generated from the PPGsignal. The plurality of features of the PPG signal can include one ormore pulse related features extracted from pulses of the PPG signal.

The plurality of features of the PPG signal can include at least one of(i) a first systolic blood pressure estimate generated using the PPGsignal as an input signal, (ii) a first diastolic blood pressureestimate generated using the PPG signal as an input signal, (iii) asecond systolic blood pressure estimate generated using a logarithmicPPG signal as an input signal where the logarithmic PPG signal can begenerated from the PPG signal, or (iv) a second diastolic blood pressureestimate generated using the logarithmic PPG signal as an input signal.The blood pressure classification can be generated using one or moredemographic features of a user of the computing device.

According to yet one other aspect, a computer readable medium caninclude computer code instructions stored thereon. The computer codeinstructions when executed by one or more processors can cause the oneor more processors to obtain one or more photoplethysmographic (PPG)signals generated from a sequence of images obtained from aphotodetector of the computing device. The one or more processors canextract a plurality of features of the PPG signal. The one or moreprocessors can generate a blood pressure classification using theplurality of features of the PPG signal and a blood pressureclassification model.

According to one aspect, a method of measuring vital signs of a user caninclude receiving a finger within a groove of a housing of a device. Thedevice can include a processor, a light source disposed within thehousing and positioned at a bottom region of the groove, and aphotodetector disposed within the housing and positioned at the bottomregion of the groove. The method can include emitting, by the lightsource, light in the groove and capturing, by the photodetector, asequence of images of the finger while the light is emitted. The methodcan include generating, by the processors, a photoplethysmographic (PPG)signal using the sequence of images to determine one or more vital signsof the user.

The method can further include determining a measurement of the one ormore vital signs using the PPG signal, and displaying the measurement ofthe one or more vital signs on a display device. The method can furtherinclude transmitting the PPG signal to a remote device. The remotedevice can determine a measurement of the one or more vital signs usingthe PPG signal and display the measurement of the one or more vitalsigns on a display device. The method can further include assessing, bythe processor, a quality of the sequence of image frames or a quality ofthe PPG signal, and providing a light signal indicative of the qualityof the sequence of image frames or the quality of the PPG signal.

The method can further include instructing the user to adjust placementof the finger within the groove, responsive to determining a poorquality of the sequence of image frames or a poor quality of the PPGsignal. The method can further include providing, via a finger groovelight pipe located at an upper edge of the groove, a light signal toprompt the user to take a measurement of the one or more vital signs.

According to one other aspect, a device for measuring blood pressure caninclude a housing including a groove to receive a finger of a user. Thegroove can have a curved end towards a center of the housing and canhave a first dimension between 20 mm and 70 mm, a second dimensionbetween 20 mm and 35 mm and a third dimension between 3 mm and 10 mm.The device can include a light source disposed within the housing andpositioned at a bottom region of the groove and configured to emit lightin the groove. The device can include a photodetector disposed withinthe housing and positioned at the bottom region of the groove andconfigured to capture a sequence of image frames while the light isemitted. The device can include a processor configured to generate aphotoplethysmographic (PPG) signal of a user using the sequence of imageframes to determine a blood pressure of the user.

The groove can extend from an edge of the housing towards a center ofthe device. The device can further include one or more tactile surfacefeatures at a first portion of the bottom region of the groove to aidthe user to position the finger such that a pulp region of the finger ispositioned above the photodetector. The photodetector can be positionedwithin the first portion and configured to capture images of the pulpregion of the finger of the user. The one or more surface features caninclude a protrusion at the bottom region and forming a closed loop toaccommodate the pulp region of the finger of the user. The first portionof the bottom region can have a first dimension between 2 mm and 10 mmand a second dimension between 5 mm and 20 mm. The thickness of theprotrusion can be less than 1 mm. The first portion of the bottom regioncan be less than 15 mm and greater than 2 mm from a center of the curvedend.

The photodetector can include at least one of a red-green-blue (RGB)photodetector, a full light spectrum photodetector, or a combination ofa RGB photodetector and an infrared photodetector. The device canfurther include a communications interface configured to communicatewith a remote device. The processor can be configured to cause thecommunication interface to transmit the PPG signal to the remote device.The remote device can be configured to determine a blood pressuremeasurement of the user using the PPG signal, and display the bloodpressure measurement on a display device.

The device can further include a display device disposed within thehousing. The processor can determine a blood pressure measurement of theuser using the PPG signal, and display the blood pressure measurement ona display device. The device can further include at least one of apressure sensor to measure pressure applied by the finger to a firstportion of the bottom region of the groove, a thermometer to measure atemperature of the finger, or an oximeter to measure oxygen level inblood flowing through the finger.

The device can further include a visual output device. The processor canbe configured to assess a quality of the sequence of image frames or aquality of the PPG signal, select, based on the quality of the sequenceof image frames or the quality of the PPG signal, from a plurality oflight outputs, a light output, and cause the visual output device toemit the light output. The device can further include a light pipedisposed within the housing at an upper edge of the groove. Theprocessor can cause the light pipe to be illuminated to prompt the userto take a measurement of the one or more vital signs.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example architecture of a computingdevice for implementing methods and processes described herein.

FIGS. 2A and 2B show diagrams illustrating placement of a fingertipagainst a camera of a mobile device, according to example embodiments.

FIGS. 3A-3D show diagrams illustrating the physics behind the use oftransdermal image data to measure blood pressure of the correspondingsubject.

FIG. 4 shows a flowchart illustrating a method of generating aphotoplethysmographic (PPG) signal using transdermal image data,according to example embodiments.

FIG. 5 shows a block diagram depicting components of a computerapplication for measuring blood pressure using transdermal image data,according to example embodiments of the current disclosure.

FIG. 6 shows plots of explained variance ratio for red (R) and green (G)frames of an image frame of a user finger.

FIG. 7 shows a diagram illustrating an example embodiment of identifyingan image region or block associated with a color frame for use ingenerating a PPG signal, according to example embodiments.

FIG. 8 shows plots of raw and filtered PPG signals, according to exampleembodiments.

FIG. 9 shows a flowchart illustrating a method of estimating bloodpressure based on a light intensity signal, according to exampleembodiments.

FIG. 10 shows a diagram depicting various processes that receive imageframes as input, according to example embodiments.

FIG. 11 is a diagram showing an example transfer function of aphotodetector, according to example embodiments.

FIG. 12A is a flowchart illustrating a methodology for calibrating aphotodetector, according to example embodiments.

FIG. 12B shows a flowchart illustrating a process for resetting oradjusting operational settings of the photodetector or camera, accordingto example embodiments.

FIG. 13 is a flowchart illustrating another methodology for calibratinga photodetector, according to example embodiments.

FIG. 14A is a flowchart illustrating a methodology for determining asequence of local quality values of a PPG signal, according to someembodiments.

FIG. 14B is a flowchart illustrating a methodology for assisting fingerplacement on the lens during camera calibration, according to someembodiments.

FIG. 15 shows a flowchart illustrating a method 1500 for calibratingexposure time and sensitivity of a photodetector or a camera, accordingto example embodiments.

FIG. 16 shows a flowchart illustrating a method for detecting presenceor absence of a finger or a body part, according to example embodiments.

FIG. 17 shows examples of individual downsampled frames or images of thesequence of downsampled images with corresponding Laplacian scores,according to example embodiments.

FIG. 18A shows a flowchart illustrating a method of determining aplacement of a finger or body part, according to example embodiments.

FIG. 18B shows a flowchart illustrating a method of computing a vectorfor assisting finger positioning of the user, according to exampleembodiments.

FIG. 18C illustrates an example of a user interface (“UI”) for assistingfinger positioning of the user, according to example embodiments.

FIG. 19 illustrates an example plot for an extraction of pulse indices,according to example embodiments.

FIG. 20A shows a flowchart illustrating a method of detecting pulses ina PPG signal, according to example embodiments.

FIG. 20B shows example subplots of extracted pairs of pulses, accordingto example embodiments.

FIG. 20C shows examples of a corrupted pulse and valid or good a pulse,according to example embodiment.

FIG. 21 shows a flowchart illustrating a method of determining one ormore error conditions based on features of pulse segments of the PPGsignal, according to example embodiments.

FIG. 22A shows plots illustrating an example PPG signal, variation inrespective pulse durations and variations in ratios of durations ofconsecutive pulses, according to example embodiments.

FIGS. 22B-G show plots illustrating an example upsampled PPG signal withextracted features and characteristics, according to exampleembodiments.

FIG. 23 shows a flowchart illustrating a method of determining perfusionof a PPG signal, according to example embodiments.

FIG. 24A illustrates an example of a negative logarithmic PPG signal (orPPG absorbance A) and the respective envelope, according to exampleembodiments.

FIGS. 24B-E show plots illustrating various logarithmic PPG signals andrespective amplitude distributions, according to example embodiments.

FIG. 25A shows a flowchart illustrating a method of detecting placementof a body part (e.g., a finger) facing a photodetector, according toexample embodiments.

FIG. 25B shows raw images captured while a user moves his finger acrossa lens of a camera, according to example embodiments.

FIGS. 25C-E show the images of FIG. 25B in the HSL color space.

FIG. 25F shows a diagram illustrating the chromatic circle.

FIG. 25G shows plots for three different penalty scores expressed asfunctions of pixel hue values, according to example embodiments.

FIG. 26 shows a flowchart illustrating a method for enhancingacquisition of transdermal optical data, according to exampleembodiments.

FIG. 27 shows a diagram illustrating an example approach of dividing animage block into a plurality of corresponding sub-regions, according toexample embodiments.

FIGS. 28A-G show plots of color signals associated with various imageblocks and different data quality conditions, according to exampleembodiments.

FIG. 29 shows a flowchart illustrating a method for assessing PPG signalquality for use to estimate blood pressure, according to exampleembodiments.

FIG. 30 shows an example diagram illustrating the use of a slidingwindow to compute the normalized autocorrelation metrics or parameters,according to example embodiments.

FIGS. 31A-B show examples of two examples of filtered PPG signals andthe corresponding normalized autocorrelation metrics or parametersq_(N)(i) for a given time window W_(N), according to exampleembodiments.

FIGS. 31C-D shows examples of peak detection in example autocorrelationarrays, according to example embodiments.

FIG. 32 shows an example image depicting individual normalizedautocorrelation parameters q_(N)(i) for a plurality of time windowsW_(N), according to example embodiments.

FIGS. 33A-B illustrate example implementations the computation ofautocorrelation coefficients, according to example embodiments.

FIG. 34 shows a flowchart illustrating a method of measuring the qualityof an acquired PPG signal, according to example embodiments.

FIGS. 35A-L are screenshots of a user interface (UI) of the computerapplication at various phases of the acquisition of transdermal opticaldata, according to example embodiments.

FIG. 36 illustrates an example of a classical offset calibration model,according to example embodiments.

FIG. 37 is block diagram illustrating an example a blood pressurecalibration system, according to example embodiments.

FIG. 38 shows an example flowchart illustrating a blood pressurecalibration method, according to example embodiments.

FIG. 39 illustrates an example of a smart offset calibration model,according to example embodiments.

FIG. 40 shows a flowchart illustrating a method of generating a bloodpressure classification based on a PPG signal and a classificationmodel, according to example embodiments.

FIG. 41 illustrates an example of an auto-calibration model, accordingto example embodiments.

FIG. 42 shows a flowchart illustrating a method of generating a bloodpressure value based on a PPG signal and an estimation model, accordingto example embodiments.

FIGS. 43A-H illustrate examples of a device for measuring vital signs ofa user in various views, according to example embodiments.

FIGS. 44A-H illustrate different views of a device for measuring vitalsigns of a user, according to example embodiments.

FIGS. 45A-H illustrate different views of a device for measuring vitalsigns of a user in broken lines, according to example embodiments.

FIGS. 46A-D illustrate perspective and top views of a device with arough surface in a groove, according to example embodiments.

DETAILED DESCRIPTION

For purposes of reading the description of the various embodimentsbelow, the following descriptions of the sections of the specificationand their respective contents may be helpful:

Section A describes an example architecture of devices and systemsconfigured to implement embodiments described in this disclosure.

Section B describes systems and methods for estimation of blood pressureusing transdermal optical recordings.

Section C describes systems and methods for calibrating a photodetectorfor measuring blood pressure.

Section D describes systems and methods for enhancing acquisition oftransdermal optical recordings for use to estimate blood pressure.

Section E describes systems and methods for assessment ofphotoplethysmographic (PPG) signal quality for initiating blood pressuremeasurement.

Section F describes systems and methods for improving an accuracy of ablood pressure measurement using offsets based on reference signalsgenerated using a measurement device.

Section G describes systems and methods for improving an accuracy of ablood pressure measurement using offsets generated without referencesignals.

Section H describes systems and methods for classifying blood pressuremeasurements instead of estimating actual blood pressure values.

Section I describes a standalone blood pressure measurement device.

Section J describes various design aspects of the standalone bloodpressure measurement device.

A. Device Architecture

FIG. 1 shows a block diagram depicting one implementation of a systemarchitecture for a computer system 100 that may be employed to implementmethods described herein, according to inventive concepts of the currentdisclosure. In general, the embodiments described in this disclosure canbe implemented by the computer system 100. The computer system 100 caninclude a single computing device including, but is not limited to, asmartphone, a tablet device, a handheld device, a smart watch, apersonal digital assistant, a mobile computing device, a laptop, adesktop, a hardware computer server, a standalone blood pressuremeasurement device, a workstation, or other type of computing device.The single computing device can be a standalone device designated formeasuring various vital signs, e.g., blood pressure, pulse rate,respiratory rate, oxygen saturation and/or body temperature, such as thedevice described in Section I. In some implementations, the computersystem 100 can include multiple computing devices that arecommunicatively connected to each other and configured to performmethods or processes described herein. For instance, the computer system100 can include a first device (e.g., a standalone device) for acquiringphotoplethysmographic (PPG) signals, and a second device (e.g., one ormore remote servers or one or more other computing devices)communicatively coupled to the first device and configured to processthe PPG signals acquired by the first device or other related signals.

In brief overview, the computer system 100 can include one or moreprocessors 102 to execute computer code instructions, a memory 104 tostore the computer code instructions and/or other data, a bus 106communicatively coupling various components of the computer system 100,a light source 108, a camera device (or a photodetector) 110 and adisplay device 112.

The one or more processors 102 can include a microprocessor, a generalpurpose processor, a multi-core processor, a digital signal processor(DSP) or a field programmable gate array (FPGA), an application-specificintegrated circuit (ASIC) or other type of processor. The one or moreprocessors 102 can be communicatively coupled to the bus 106 forprocessing information. The memory 104 can include a main memory device,such as a random-access memory (RAM) or other dynamic storage device,coupled to the bus 106 for storing information and instructions to beexecuted by the processor 102. The main memory device can be used forstoring temporary variables or other intermediate information duringexecution of instructions (e.g., related to methods described herein oroperations thereof) by the processor 102. The computer system 100 caninclude a read-only memory (ROM) or other static storage device coupledto the bus 106 for storing static information and instructions for theprocessor 102. The ROM can store computer code instructions related to,or representing an implementation of, methods described herein, such asapplication 114. A storage device, such as a solid state device,magnetic disk or optical disk, can be coupled to the bus 106 for storing(or providing as input) information and/or instructions.

The computer system 100 can include, or can be communicatively coupledto, the light source 108, the camera device 110 and the display device112. The one or more processors 102 and memory 104 can becommunicatively coupled to the light source 108, the camera device 110and the display device 112 via the bus 106. The light source 108 andcamera device 110 can be positioned on a front side of the computersystem 100 or on back side of the computer system 100. The light source108 can be a flash device associated with the camera device 110. In someimplementations, the computer system 100 can include multiple cameradevices 110, such as one or more front cameras and one or more backcameras. In some embodiments, the camera device 110 may be a separatedevice that can be communicatively coupled to the computer system 100.The one or more processors 102 can control the light source and thecamera device 110. For instance, the one or more processors 102 canautomatically or upon user input, trigger the light source 108 to emitlight and trigger the camera device 110 to capture images or a sequenceof image frames. The one or more processors 102 can control settings ofthe light source 108 and the camera device 110. For instance, the one ormore processors 102 can adjust one or more settings of the camera device110 prior to or while capturing image frames.

The computer system 100 can include other input devices (e.g., otherthan the camera device 110), such as a keyboard including alphanumericand other keys, a touch screen, or a communication interface forreceiving input data. The input devices can be communicatively coupledto the bus 106 for communicating information and command selections tothe one or more processors 102, the memory 104 and/or the camera device110. The input devices can include a cursor control, such as a mouse, atrackball, or cursor direction keys, for communicating information andcommand selections to the one or more processors 102, the memory 104and/or the camera device 110.

The display device 112 can include a Liquid Crystal Display (LCD),Thin-Film-Transistor LCD (TFT), an Organic Light Emitting Diode (OLED)display, LED display, Electronic Paper display, Plasma Display Panel(PDP), or other display, for displaying information to a user of thecomputer system 100. The display device 112 can include a touch screencapable of acting as an input/output (I/O) device. The computer system100 can include other output devices (e.g., other than the displaydevice 112) such as a communication interface (not shown in FIG. 1 ) forcommunicating information to other external devices. The communicationinterface can include a wired communication interface, a wirelesscommunication interface, BLUETOOTH, near field communication (NFC)interface, other communication interface or a combination thereof.

According to various implementations, the methods described herein orrespective operations can be implemented as an arrangement of computercode instructions that are executed by the one or more processors 102 ofthe computer system 100. The arrangement of computer code instructionscan be implemented as client application 114. The client application 114can be installed on, and executed by, the computer system 100 to performembodiments described herein. The client application 114 can includemodules, instructions and/or application programming interfaces (APIs)for executing various operations related to embodiments describedherein, such as triggering the light source 108 to emit light,triggering the camera device 110 to capture images, controlling settingsof the camera device 110, displaying user interfaces (UIs) on thedisplay device 112, processing image data, generatingphotoplethysmographic (PPG) signals based on image data, processingand/or assessing the quality of PPG signals, computing blood pressureand/or pulse measurements or a combination thereof. The computer codeinstructions and/or data associated with the application 114 can bestored in ROM or the storage device of the computer system 100, and readinto a main memory device (e.g., RAM) of the computing device, forexample, when the application is initiated.

In some implementations, the application 114 can include a clientapplication to run on the computer system 100 and a server applicationto run on a cloud server or other remote computer server. The serverapplication can be a web application. The remote server and the serverapplication running thereon can be configured to serve a plurality ofcomputing devices 100 that can connect to the server via a communicationnetwork. In some implementations, the client application and thecomputer system 100 can perform some of the operations associated withembodiments described herein, while the server and the serverapplication can execute other operations of the embodiments. Forinstance, the computer system 100 and the client application 114 canperform image data acquisition, PPG signal generation and/or display ofoutput results, while the server and the server application can computeblood pressure and/or pulse measurements based on PPG signals receivedfrom the computer system 100.

In some other implementations, hard-wired circuitry may be used in placeof or in combination with software instructions to effect illustrativeimplementation of the methods described herein or operations thereof. Ingeneral, implementations are not limited to any specific combination ofhardware circuitry and software. The functional operations described inthe following sections can be implemented in other types of digitalelectronic circuitry, in computer software, firmware, hardware or acombination thereof.

B. Estimation of Blood Pressure Using Transdermal Optical Recordings

The most common non-invasive approach for measuring blood pressure makesuse of the sphygmomanometer; a device that includes an inflatable cuffand a configured to surround the arm of a patient or subject and astethoscope. A user inflates the cuff above an expected systolic bloodpressure (SBP) level and then releases the pressure inside the cuff. Thestethoscope is used to listen to the blood circulating in the arteries.When the blood pressure decreases below the SBP level, blood flowturbulence generates noise known as Korotkoff noise that is audibleuntil the blood pressure gets below the diastolic blood pressure (DBP)level. As such, the SBP is determined as the recorded pressure at thestart of the Korotkoff noise, and the DBP is determined as the recordedpressure at the end of the Korotkoff noise. While it provides accurateblood pressure measurements, the sphygmomanometer may not be easy andcomfortable to use. For instance, many patients need the help of anotherperson to place the cuff on the patient's arm and operate thesphygmomanometer. Also, the sphygmomanometer provides singlemeasurements at specific time instances, but does not allow forcontinuous monitoring of blood pressure or for analyzing blood pressurewaveforms over time.

A more advanced noninvasive approach for measuring blood pressureincludes the use of oscillometric cuffs, which operate in a similar wayas sphygmomanometers except for the use of the stethoscope to listen tothe blood circulating in the arteries. Instead, the oscillometric cuffincludes an auto-inflatable cuff that is linked to a microphone formonitoring the Korotkoff noise and deducing the SBP and DBP levels basedon the Korotkoff noise. Oscillometric cuffs are easier and moreconvenient to use to monitor blood pressure on a regular basis,especially for hypertensive patients. However, the oscillometric cuffsstill need to be placed around the patient's arm to apply a pressurethereon, and are usually less accurate than sphygmomanometers.Furthermore, the oscillometric cuffs are not so cheap and are notconvenient to carry around.

Another approach is to measure intra-arterial blood pressure using acatheter. This approach allows for continuous recording and monitoringof blood pressure over time, and provides other indicators such as gasconcentration in the blood, which is useful to measure bloodoxygenation. However, this is an invasive approach that requires a heavyequipment to prevent the patient from moving. Also, the procedure issignificantly uncomfortable for patients, and is usually available forpatient in relatively sever conditions.

In the current disclosure, a simpler and easier to use noninvasiveapproach for measuring blood pressure makes use of transdermal opticaldata recorded by a camera 110 of a computer system 100, such as a smartphone, tablet, Internet-of-things (IoT) device, or other deviceconfigured or structured to measure blood pressure. The computer system100 can leverage the non-invasive technique of photo plethysmography(PPG) to estimate the blood pressure of a user or subject. Specifically,upon the user placing his fingertip (or other surface of the body) toface the lens of the camera, the camera 110 can capture a sequence ofimages of the fingertip. The computer system 100 can generate a colorsignal representing a PPG signal using the sequence of images, and usethe generated PPG signal to determine blood pressure measurements and/orpulse measurements.

FIGS. 2A and 2B show diagrams illustrating the placement of a subject'sfinger to a camera 204 of a mobile phone 202, according to inventiveconcepts of the current disclosure. FIG. 2A shows a rear side of themobile phone 202 with a rear-facing camera 204. The rear-facing camera202 can include a lens (not shown in FIGS. 2A and 2B) and an optical (orlight) waveguide to channel light towards the camera lens. The opticalwaveguide can include a recess or groove in the rear side or surface ofthe mobile phone 202. The mobile phone 202 can include a transparentglass (or other material) arranged at the upper end of the opticalwaveguide, e.g., aligned with the rear surface of the mobile phone 202.The mobile phone 202 can include a flash device (or light source) 206arranged adjacent to the camera 204 or the glass at the upper end of theoptical waveguide. The flash device 206 can automatically produceartificial light to help illuminate a scene at the time of capturingimages or image frames by the camera 204. In some implementations, theflash device 206 can be positioned adjacent to the camera 204, e.g.,within or at the bottom of the optical waveguide. FIG. 2B shows thefront side of the mobile phone 202 with the camera 202 facing downward.The smartphone 202 represents an example implementation of the computersystem 100, and the camera 204 represents an example implementation ofthe camera device 110. Other devices can be used, instead of the mobilephone 202, to capture image frames and use the captured image frames tomeasure blood pressure and/or other vital signs. Also, while FIGS. 2Aand 2B show a finger is being used, in general other body parts can beplaced in front of the camera lens to capture image frames for thepurpose of measuring blood pressure or other vital signs.

The user can place or apply their fingertip, or other body part, againstthe camera lens, the optical waveguide of the camera 204 or against thetransparent glass arranged at the upper end of the optical waveguide. Insome implementations, the user can place their fingertip or other bodypart to face, or in front of, the lens of the camera. The fingertip orthe other body part can be placed to be exposed (e.g., adjacent) to theflash device 206. Considering FIG. 2B, the user can place theirfingertip or other body part against the camera lens, the opticalwaveguide or the transparent glass covering the optical waveguide whilethe screen of the mobile device 202 is facing the user. Accordingly, theuser can activate the application 114 to trigger the camera 204 tocapture a sequence of image frames, and use the image data to estimateblood pressure of the user. From the user's perspective, the procedureis simple, easy, and can be carried out anywhere as long as the user isequipped with the smartphone 202 or other computer system 100 having arespective light source 108, a camera (or photodetector) 110 andinstalling the application 114.

FIGS. 3A-3D are diagrams illustrating the physics behind the use oftransdermal image data to measure blood pressure of the correspondingsubject. The diagrams in FIGS. 3A-3C illustrates propagation of lightwhen a finger 302 or other body part is applied to a camera 204 ofsmartphone 202 (or other device). Light rays 304 emitted by the flashdevice (or light source) 206 penetrate the finger 302. Specifically,incident light 304 penetrates the tissue as well as the blood vessels306 of the finger 302 or other body part. The blood circulating in theblood vessels 306 includes arterial blood and venous blood, and thelight rays 304 penetrates both the arterial blood and the venous blood.The flow of arterial blood is pulsatile, or following a pulse wave 308caused by extension of the intramural coronary vessels due to additionalpressure consequent to contraction of the cardiac muscle.

A first portion of the incident light 304 gets absorbed by the bloodwhile a second portion 310 gets reflected back from the finger 302, andis received by the red-green-blue (RGB) sensor 312 of the camera 204.The light intensity received by the RGB sensor 312 can be described as:

I=I ₀·10^(−A) =I ₀·10^(−Σ) ^(i) ^(a) ^(i) ^(l) ^(i) ,  (1)

where I₀ represents the light intensity 304 produced by the light source206, I represents the light intensity 310 received by the RGB sensor (orphotodetector) 312, A represents light absorbance by the finger 302 (orother body part) applied to the camera 204, the coefficient a_(i)represents a linear absorption coefficient of tissue i and l_(t)represents the thickness of tissue i. The reflected light intensity Irepresents a transdermal optical signal of the subject or user. Thecamera 204 can use the reflected light intensity 1310 received at eachtime instance by the RGB sensor 312 to generate a corresponding image.Specifically, the camera 204 can generate a sequence of image framesrepresenting the light intensity I 310 received at a plurality of timeinstances. For instance, the camera 204 can generate the image frames ata frame rate of 25 frames per second, at 30 frames per second or atother frame rate.

FIG. 3D shows a diagram illustrating various light absorption componentsassociated with the finger 302 or other body parts. Light for a givenwavelength is absorbed by the different layers of flesh of the finger.Tissue and venous blood absorb a constant proportion of light throughtime while pulsatile arterial blood has a varying volume and thus has avarying absorption through time. Specifically the light absorbance A canbe described as:

A=a _(flesh) ·l _(flesh) + . . . +a _(veinous) ·l _(veinous) +a_(arterial) ·l _(arterial)(t)=A ₀ +a _(arterial) ·l _(arterial)(t).  (2)

The term a_(flesh)·l_(flesh) represents the light absorbance of flesh,a_(veinous)·l_(veinous) represent the light absorbance by the veins, anda_(arterial)·l_(arterial)(t) represents the light absorbance bypulsatile arterial blood. The term A₀ represents the aggregatenon-pulsatile light absorbance by the finger (or body part) 302. For thepurpose of measuring blood pressure and/or pulse of the subject, thefocus is on the component of the reflected light intensity I 304 definedin terms of a_(arterial)·l_(arterial)(t), e.g., as I₀e^(−a) ^(arterial)^(·l) ^(arterial) ^((t)), because it is time varying (AC signal) and isexpected to reflect the pulse wave 308. Other useful metrics that aredefined herein to estimate the proportion of light absorbed by a system,include the transmittance T that measures the proportion (or ratio) oflight transmitted or reflected by a system (e.g., finger 302) relativeto incident light I₀ or light I₀ produced by light source 108 or 206.That is,

$\begin{matrix}{T = {\frac{I}{I_{0}}.}} & (3)\end{matrix}$

Another metric is opacity which measures the proportion (or ratio) oflight that is absorbed:

$\begin{matrix}{O = {\frac{I}{T}.}} & (4)\end{matrix}$

Referring to FIG. 4 , a flowchart illustrating a method 400 ofgenerating a PPG signal, for measuring blood pressure, using transdermalimage data is shown, according to inventive concepts of this disclosure.The method 400 can include acquiring a sequence of images representingtransdermal optical data of a subject (STEP 402), and generating asequence of downsampled image frames using the acquired sequence ofimages (STEP 404). The method 400 can include identifying an imageregion across the sequence of downsampled image frames (STEP 406), andgenerating a PPG signal associated with the image region across thesequence of downsampled images (STEP 408). The method 400 can beperformed by the computer system 100 or the corresponding processor 102,for example, upon executing the application 114.

Referring to FIG. 5 , a block diagram describing components of theapplication 114 is shown, according to example embodiments of thecurrent disclosure. The application 114 can include a data acquisitionmodule 502 to acquire image data, and a processing module 504 to processthe image data. The application 114 can include a signal-generatingmodule 506 to generate a PPG signal using the image data, and a signalquality assessment module 508 to assess the quality or validity of thePPG signal. The application 114 can include an optical blood pressuremonitoring (OBPM) module 510 to estimate blood pressure measurementsusing the PPG signal, and an output module 512 to output or displayblood pressure measurements. The application 114 can include a cameracalibration module 514 for calibrating the camera or photodetector 110,and a calibration/estimation module 516 for enhancing the accuracy ofblood pressure measurements. The functions associated with these modulesor components are discussed in further detail below.

Referring back to FIGS. 1-5 , the method 400 can include the computersystem 100 acquiring a sequence of images representing transdermaloptical data of a subject (STEP 402). A user can actuate the application114 on the computer system 100, and apply their finger pulp or otherbody part to the camera 110 or photodetector 312, for example, asillustrated in FIGS. 2A, 2B and 3A-3C. Upon initiation, the application114 can cause the computer system 100 to display a user interface (UI)on the display device 112. The UI can provide instructions to the userto apply their finger pulp against the camera 110 or photodetector 312(or against a light/optical waveguide of the camera) for a given periodof time, e.g., 30 seconds. The UI can provide an option, e.g., upon userselection, to present further instructions, images or a demo explaininghow the finger pulp (or other body part) is to be applied to the camera110.

The application 114 or the data acquisition module 502 can automaticallytrigger the camera 110 to acquire a sequence of image frames. In someimplementations, the application 114 or the data acquisition module 502can cause the computer system 100 to display an interactive item as partof the UI. The application 114 or the data acquisition module 502 canautomatically trigger the camera device 110 to acquire the sequence ofimages, upon the user interacting with the interactive item. Triggeringthe camera device 110 can include actuating the flash device 206 to emitartificial light either continuously or periodically according to agiven frequency.

The camera device 110 can acquire the sequence of image frames while theuser's finger pulp (or other body part) is applied against the opticalwaveguide of the camera or the corresponding transparent glass.Specifically, the RGB sensor 312 can acquire the sequence of imageframes according to a specific or predefined frame rate based on thelight intensity I_(r) reflected from the user's finger. The frame ratecan be equal to 25 frames/second (fps), 30 fps, 50 fps or other framerate supported by the camera device 110. The data acquisition module 502can receive image data corresponding to image frames from the cameradevice 110. In some implementations, the data acquisition module 502 canreceive image data for each frame immediately after the image frame iscaptured or recorded by the RGB sensor 312 or the camera device 110.

In some implementations, the computer system 100 can include a pluralityof cameras (or photodetectors) 110. The computer system 100 canautomatically select one of the cameras (or photodetectors) 110 toacquire the sequence of image frames, and display (e.g., via a userinterface of the application 114) a notification to the user of thecomputer system 100 of the selected camera 110. The user can placehis/her finger against the selected camera. The selection can be basedon features or characteristics of the various cameras. In someimplementations, the user can select the camera to be used to acquirethe sequence of image frames. The computer system 100 can receive orobtain the user selection via an input device, e.g., a keyboard, a mouseor a touch screen, among others, of the computer system 100. Thecomputer system 100 may confirm the user selection via, for example, thedisplay device 112. The computer application 114 may include differentsettings for distinct cameras 110. The computer system 100 can maintaincamera ID/phone model matrix in the memory 104. Besides the camera ID,the computer system 100 can store the preferred camera settings in thematrix, such as ISO, exposure time, tonemap, RGB gain or a combinationthereof, among other settings.

The method 400 can include the computer system 100 generating a sequenceof downsampled image frames (STEP 404), and identifying an image regionor an image block across the sequence of downsampled image frames (STEP406). The camera device 110 can generate, for each frame, an imagehaving several thousands of pixels. This information is often noisy andredundant. The pixels provide essentially the same information aboutblood flow in finger arteries since they capture light transmitted byvery close regions of finger flesh. In other words, the image framescaptured by the camera 110 are not expected to have much contrast, edgesor other visual details of interest. The relevant data here is thevariation in cumulative color intensity of the captured image frames.

Principal component analysis (PCA) for an image frame reveals how toreduce the dimensionality of the data samples and find out whichaggregation function best captures all information in the sequence ofcaptured image frames or signals. FIG. 6 shows a graph 600 illustratingplots 602 and 604 of the explained variance ratio for the red (R) andgreen (G) frames of an image frame of a user finger, respectively. Thex-axis represents the component number or index, and the y-axisrepresents the explained variance ratio. For each component, thecorresponding variance ratio represents the variance of the componentdivided by the total variance of the color frame. Both the red plot 602and the green plot 602 in FIG. 6 show that almost all the variance ofthe signal is contained in a single component, i.e., the firstcomponent. The first component captures more than 99% of the color(e.g., R or G) frame variance. This decomposition is strikingly close toa simple averaging—give or take a multiplicative factor—of the pixels ineach color (e.g., R or G) frame of the sequence of image frames.

The data acquisition module 502 or the processing module 504 candecompose each image frame into R, G and blue (B) frames. For thepurpose of estimating blood pressure and/or pulse of the user, theapplication 114 or the computer system 100 can use the R frames or the Gframes. In some implementation, the application 114 or the computersystem 100 may use the B frames for estimating blood pressure or pulse.In some implementations, the application 114 or the computer system 100may use a combination of R, G and/or B frames. In some implementations,the application 114 or the computer system 100 may use image framesassociated with another color space (or color model), e.g., other thanthe RGB color space (or RGB color model), such as the YUV color space orthe cyan, magenta, yellow (CMY) color space.

FIG. 7 shows a diagram illustrating an example embodiment ofdownsampling the sequence of image frames and identifying an imageregion or block associated with a color (e.g. G or R) frame 702. Theprocessing module 504 can downsample each color (e.g. G or R) frame 702of the sequence of image frames to a corresponding downsampled imageframe 704. The processing module 504 can downsample the color (e.g. G orR) frame 702 to a 5×5 image (or data) block, a 9×9 image (or data)block, or an image block 704 of a different size. In downsampling thecolor frame 702, the processing module 504 can apply low-pass filteringor averaging to the pixel values of the original color frame 702. Theprocessing module 504 can downsample the color frame 702 using knowndownsampling techniques.

The processing module 504 can identify a sub-block 706 of the imageblock 704 for use to generate a color signal. The processing module 504can identify a central zone or sub-block of the image block 704 assub-block 706. Depending on the size and the placement of the userfinger against the camera, the color intensities of a central zone ofthe image block 704 (or the color frame 702) are more likely to reflectthe pulsatile nature of arterial blood in the finger than outer regionsof the image block 704 (or the color frame 702). In some cases, the userfinger may not be placed right at the center of the camera device 110,but rather shifted towards one direction over another. In such cases, atleast one outer region of the image frame may not adequately reflect thelight (or optical signal reflected from the user finger. In general, thedownsampled image block 704 can have an n×n size while the sub-block 706can have an m×m size, where n and m are integers and n is greater thanm. In some implementations, n=5 and m=3, or n=9 and m=5. In someimplementations, the downsampled image block 704 and/or the sub-block706 can have rectangular shape.

Referring back to FIG. 4 , the method 400 can include the computersystem 100 generating a PPG signal, for use to determine a bloodpressure of the subject, using the image region or image blockidentified across the sequence of downsampled image frames (STEP 408).For each color (e.g., R or G) frame of the sequence of image frames, thesignal-generating module 506 can average the pixel intensities of thecorresponding sub-block 706 to generate a single value for thecorresponding image frame. Repeating the averaging process for eachimage frame leads to a one dimensional color intensity signalrepresenting the PPG signal. Considering, the nature of the informationin the image frames and the PCA results discussed above with regard toFIG. 6 , the averaging of the pixel intensities of the each sub-block706 is a good way to capture essentially the variation in lightintensity I over time. In some implementations, the signal-generatingmodule 506 may apply a weighted averaging to the pixel intensities ofeach sub-block 706.

The signal-generating module 506 can filter the color or PPG signalusing a high-pass filter to remove DC or non-pulsatile components of thesignal. Specifically, referring back to FIG. 3B, the high-pass filteringcan remove components of the color or PPG signal representing lightreflections from the finger tissue, venous and capillary blood andnon-pulsatile arterial blood. The high-pass filtered signal canrepresent essentially the light reflected from pulsatile arterial blood.In some implementations, the signal-generating module 506 can apply ahigh-pass filter with cut-off frequency between 0.4 Hz and 4 Hz to thegenerated color or PPG signal. For instance, the cut-off frequency canbe equal to, but not limited to, 0.4 Hz, 0.5 Hz, 0.6 Hz, 0.7 Hz, 0.8 Hz,0.9 Hz, 1 Hz, 1.1 Hz, 1.2 Hz, 1.3 Hz, 1.4 Hz, 1.5 Hz, 1.6 Hz, 1.7 Hz,1.8 Hz, 1.9 Hz, 2 Hz, 2.1 Hz, 2.2 Hz, 2.3 Hz, 2.4 Hz, 2.5 Hz, 2.6 Hz,2.7 Hz, 2.8 Hz, 2.9 Hz, 3 Hz, 3.1 Hz, 3.2 Hz, 3.3 Hz, 3.4 Hz, 3.5 Hz,3.6 Hz, 3.7 Hz, 3.8 Hz, 3.9 Hz or 4 Hz, among others. In someimplementations, the cut-off frequency can be determined based ontraining PPG data.

FIG. 8 shows a graph 800 illustrating plots of raw and filtered PPGsignals, according to inventive concepts of the current disclosure. Theplot 802 represents a raw color or PPG signal generated by averagingpixel values for each sub-block 706. The plot 804 represents thehigh-pass filtered PPG signal corresponding to the raw PPG signal 802.The plot 806 represents the low-pass component of the PPG signal 802 (orcolor intensity signal) removed by the high-pass filter. The high-passfiltered signal 804 shows a pulse waveform that reflects better (e.g.,compared to the raw PPG signal 802) the pulse of the user.

In a way, the computer system 100 or application 114 can reduce eachimage frame captured by the camera device 110 to a single numericalvalue equal to the sum (or weighted sum) of the pixel values of thecorresponding sub-block 706. Such single numerical value can be viewedas representing an estimate of the intensity of the light reflected fromthe finger (or other body part) at the time the image frame wascaptured. The signal-generating module 506 can stack the numericalvalues for a given color channel (e.g., R or G) to generate the PPGsignal, and filter the PPG signal using the high-pass filter toeliminate DC or non-pulsatile signal components. The PPG signal 802 canbe viewed as a sequence of color intensity values each of whichgenerated from a corresponding image frame or a corresponding colorframe of the acquired sequence of images.

The method 400 can further include determining a blood pressuremeasurement of the subject using the PPG signal. The optical bloodpressure monitoring (OBPM) module 510 can implement an OBPM algorithm toestimate or compute systolic and diastolic blood pressure measurementsof the subject, using the PPG signal. The OBPM module 510 can receivethe PPG signal or the corresponding logarithmic PPG signal to estimateor compute the systolic and diastolic blood pressure measurements of theuser. The OBPM module 510 can estimate or compute the heart rate orpulse rate of the user, using the PPG signal. The pulse rate can beestimated by identifying pulses of the PPG signal and computing theinverse of pulse duration or counting the number of pulses per second.The OBPM module 510 can estimate or compute the mean blood pressure orother blood pressure features of the user, using the PPG signal.

The OBPM algorithm is based on pulse wave analysis (PWA), for example,as discussed in Chapter 8 of the book entitled “The Handbook of CufflessBlood Pressure Monitoring A Practical Guide For Clinicians, Researchersand Engineers” by Josep Sold and Ricard Delgado-Gonzalo, Springer NatureSwitzerland AG 2019. The OBPM module 510 can employ the oBPM™approach/technology described in the International Patent PublicationNo. WO2016138965A1 and in “Measuring Pressure: Introducing oBPM, theOptical Revolution for Blood Pressure Monitoring” IEEE Pulse (Volume: 9,Issue: 5, September-October 2018).

In some implementations, the computer system 100 or thecalibration/estimation module 516 can use outputs of the OBPM algorithmand/or other features to determine blood pressuremeasurements/estimates, as discussed in further detail in sections F andG below. In some implementations, the computer system 100 or thecalibration/estimation module 516 can use outputs of the OBPM algorithmas well as other features to determine a classification of the bloodpressure of the subject, as discussed in further detail in section Hbelow

The output module 512 can cause the computed blood pressure measurementand/or other measurements to be presented to the user on the displaydevice 112. The output module 512 can also manage the rendering of UI(s)of the application 114 and the rendering of data or content associatedwith such UI(s). For instance, while the camera device 110 is stillacquiring image frames, the output module 512 can render a UI or contentindicating that the blood pressure measuring process is still going onso that the user keeps his finger (or other body part) applied to thecamera device 110. The output module 512 can render instructions, imagesor demos to explain to the user how to place the finger (or other bodypart) against the camera device 110 or how to access (or use) variousfeatures of the application 114. The output module 512 can cause thecomputer system 100 to transmit measured blood pressure values or othermeasurements to another device or to the cloud via a communicationnetwork.

Estimating blood pressure based on a sequence of RGB images can involvemore processes or steps than those described in relation to FIG. 4 .Referring to FIG. 9 , a flowchart illustrating a method 900 forestimating blood pressure based on RGB image data is shown, according toexample embodiments. The method 900 can be implemented in theapplication 114, and can be executed by a single device, such ascomputer system 100, or by more than one device, such as a mobile deviceand remote server, a standalone device and a mobile device or othercombinations of devices. Accordingly, the application 114 can beimplemented as a single application executing on a single device or asmultiple applications executing on different devices to performdifferent processes or steps of method 900.

The method 900 can include initiating (e.g., via application 114) theblood pressure measuring process (STEP 902), and the computer system 100opening or activating the camera responsive to the initiation of theblood pressure measuring process (STEP 804). The computer system 100 cancalibrate the camera (or photodetector) 110, 204 or 312 (STEP 906)before starting the acquisition of image frames. The camera calibrationcan be timed. If the camera calibration fails within a predefinedmaximum time duration for camera calibration, the computer system 100can provide an alert signal (e.g., for display on display device 112) toindicate the failure of camera calibration (STEP 908). If the cameracalibration process is successful within the predefined maximum timeduration for camera calibration, the computer system 100 can initiate adata acquisition timer (STEP 910), and start the acquisition of imageframes for use to measure blood pressure and/or other vital signs (STEP912). During the data acquisition process, the computing device canassess the quality of the acquired data, and may provide feedback (e.g.,via display device 112) regarding the quality of the acquired data. Ifthe quality of the PPG signal or a portion thereof is determined to bepoor or unsatisfactory, the computer system 100 can provide an alertsignal/message regarding the quality (STEP 914). The computer system 100can detect placement of the finger (or body part) over the camera or thecorresponding lens or light waveguide, and output alert signals/messagesregarding the finger or body part placement (STEP 916). If a PPG signalof a predefined duration (e.g., 30 seconds) that is generated from asequence of image frames is determined to have a “good quality” (e.g.,based on some quality assessment metrics or procedures), the computersystem 100 can terminate the data acquisition timer and close/deactivatethe camera 110 (STEP 918), and generate an estimate or a classificationof the blood pressure and/or other vital signs based on the PPG signal(STEP 920). If the computer system 100 is unable to generate theestimate of the blood pressure, it can provide an alert or error message(STEP 922). Otherwise, the computer system 100 can determine the bloodpressure estimation process to be a success (STEP 924), and can outputthe blood pressure estimate or classification on the display device 112.

Referring to FIG. 10 , a diagram 1000 depicting various processes thatinvolve access to the camera are shown, according to exampleembodiments. Processes such as the camera calibration process 1002, thesignal quality assessment (autocorrelation) process 1004 and the PPGsignal generation process 1006 receive image frames as input from thecamera 110 and may assess signal quality on a frame by frame basis.

Processes described in FIGS. 9 and 10 , and other processes implementedwithin application 104 or within another component of a blood pressureestimation system, are described in further detail in the followingsections.

C. Camera Calibration

A photodetector, also referred to as a photosensor, is a sensor of lightor other electromagnetic radiation that can transform a continuousphysical value, e.g., light intensity, into a discrete numerical value,e.g., a pixel value. A photodetector has a gain that represents therelationship between the number of electrons acquired on an image sensorand the number of photoelectrons measured at the end of exposure (e.g.,analog-to-digital units (ADUs)). As a result of increasing the gain, theimage signal can be amplified and the apparent brightness of an image ata given exposure can be improved.

One problem relates to calibrating the photodetector (or camera) 110 isto determine an optimal or adequate light intensity range associatedwith an approximatively linear portion of a transfer function of thephotodetector 110. FIG. 11 shows a diagram 1100 showing example transferfunctions of a photodetector, representing a relationship between photoninputs (e.g., incident physical light) and electron outputs (e.g.,measured light). FIG. 11 shows three different transfer functionsincluding (1) an ideal transfer function 1102 with no quantization, (2)a transfer function 1104 depicting the effect of quantization, and (3) atransfer function 1106 depicting the effect of both quantization andnoise.

If there were no quantization nor any noise, the transfer function wouldbe perfectly linear (e.g., the identity function or a multiplicativescaling) as is the case for the transfer function 1102, which would bean ideal scenario in the context of blood pressure (or other vitalsigns) measurement. A linear function would simply add a constant and/orscale incident light by a multiplicative factor, but would not changethe shape of the resulting or corresponding PPG signal. Quantization isa nonlinear operation or transformation, which introduces nonlineardistortions into the measured signal (e.g., pixel intensities) comparedto the light intensity I reflected from the finger or body part. Even inthe absence of noise, the nonlinear distortions due to quantization canbe significant especially for relatively low light intensity values. Asdepicted in the transfer function 1104, the quantization transferfunction is a stair-like function. Since the quantization error isusually constant, the lower the measured light intensity the lower isthe signal-to-noise ratio (SNR) of (or the more significant is theeffect of the quantization on) the recorded pixel values. Also, lightintensity values that are relatively high may lead to pixel saturation,where the pixel values are capped to a predefined threshold as depictedin the upper end of transfer function 1104. These effects ofquantization at the lower and upper ends of the transfer function 1104can significantly degrade the SNR of the generated PPG signal. However,in the central or middle region of the transfer function 1104, theeffect of quantization is less significant because the quantizationerror is significantly smaller than the measured light intensities.Also, pixel saturation is less likely to happen when the measuredcumulative light intensity (e.g., color intensity values forming the PPGsignal) is limited or restricted to a medium or central range.

Besides quantization, noise adds to the degradation of the SNR of pixelintensities as well as the SNR of the PPG signal as depicted in thetransfer function 1106, which depicts a cumulative effect ofquantization and noise. The transfer function 1106 represents a morerealistic transfer function because cameras usually apply quantizationand suffer from noise (e.g., additive noise). Accordingly, it isdesirable to limit measured intensity values to a middle range, such asthe range 1120, and avoid the lower range 1110 and the upper range 1130so that the transfer function (e.g., transfer function 1106) of thephotodetector is approximately linear. Given that the PPG signal isgenerated by averaging pixel intensity valued of image blocks 706 withindownsampled image or color frames, limiting the measured intensityvalues to a middle range away from lower and upper ends of the transferfunction (e.g., transfer function 1106) can reduce the likelihood ofsignificant artifacts due to camera hardware and increase the SNR of thegenerated PPG signal.

To solve these problems, according to certain aspects, embodiments inthe present disclosure relate to techniques for maintaining good qualityPPG signals by reducing the likelihood of artifacts due to camerahardware. As described above, a PPG signal may be generated by averagingpixel values of sub-blocks of downsampled image frames (e.g., sub-block706 of color image frame 702 in FIG. 7 ). Accordingly, the PPG signalvalues are in the range of pixel values (for example, the range of (0,255) if a pixel value is stored in an 8-bit integer). In someembodiments, in order to maintain good quality PPG signals, a system forcalibrating a camera device 110 can calibrate the camera gain tomaintain PPG signal values within a middle range 1120 that correspondsto an approximately linear portion of the transfer function (e.g.,transfer function 1106) of the photodetector or a camera 110. Suchcalibration ensures that the transformation of the incident light signalto a corresponding digital signal is as close to a linear transformationas possible. For example, the computer system 100 can calibrate thephotodetector (or camera) gain so that values of the PPG signal arebetween 100 and 150. In some implementations, other ranges can be usedsuch as 90 to 160, 95 to 155 or 105 to 145. In some implementations, acamera calibration process for blood pressure measurement can adjustcamera gain so that at the generated PPG signal values are within arange around (or centered at) 128 or around (or centered at) a valueclose to 128.

According to certain aspects, embodiments in the present disclosurerelate to techniques for allowing a camera calibration process to havetwo possible output states that are mutually incompatible and reachablein a finite time—(1) a ready-to-record state of starting a recording,and (2) an error state of calibration timeout. If the camera calibrationprocess is successful (e.g., successful in adjusting or setting thecamera gain to enforce PPG values to be within a target range and/ordetecting a pulsatile of the user, etc.), the process can includeoutputting the ready-to-record state to start the recording. If thecamera calibration process is not successful, the computer system 100can output the error state to raise a calibration timeout.

According to certain aspects, embodiments in the present disclosurerelate to techniques for configuring settings of a photodetector (or acamera) to ensure that the photodetector or camera should be compatiblewith predetermined requirements. For example, camera settings areconfigured to ensure that (1) measured numerical values are as close tounderlying physical signal as possible and that (2) a user is able totake measurements in good conditions for PPG signals to be of goodquality for measuring blood pressure.

In some embodiments, a camera calibration process may start with initialsettings of camera parameters. For example, the gains can be set usingpreviously saved settings (e.g., exposure time, sensitivity, etc.) anddefault values for others. In some embodiments, camera parameters can beset according to the following rules: (1) the camera is able to take atleast a predetermined number of frames per second (e.g., 25 frames persecond or more); (2) an output digital value for the pixels is as closeto a linear transform of an input light intensity as possible; (3)recorded PPG input signals can evolve or vary in a range of values thatis still in a linear response domain while having a wide spread, e.g., avalue that is not too high nor too low; (4) flash is enabled; (5) nocolor correction is applied, or alternatively such transformation isapplied in a respective single color channel and is linear (or close tolinear), e.g., color channel by color channel, so that a transformationmatrix is diagonal; and/or (6) no autocorrection is applied, or allsettings are defined manually by an application (e.g., an app in amobile device).

According to certain aspects, embodiments in the present disclosurerelate to techniques for performing a process of updating an exposurevalue (EV) to set PPG signals to be in an appropriate range of values(e.g., between 100 and 150). For example, at the end of the EV updateprocess, PPG signals can be in an appropriate range of values (e.g.,around 128 out of the full range (0, 255)) if the calibration issuccessful, or close to the appropriate range even if the calibration isnot successful.

In some embodiments, the camera calibration process can update the gainsto set a PPG signal in an acceptable range (e.g., a linear responsedomain) by updating an exposure value (EV) of the camera. If the EVupdate is successful in setting the camera gain to set PPG values in atarget range, the camera calibration process can finalize or end the EVupdate. If the EV update is not successful, the camera calibrationprocess can continue the EV update until PPG values lie in the targetrange.

In some embodiments, for updating EV, the camera calibration process mayrecord a color intensity (or PPG) signal for a predetermined orpredefined duration (e.g., 0.5 seconds of image frames) such that anongoing capture request can have time to be built and gains can beindeed updated. The predetermined or predefined duration allows for anyEV or sensitivity update that was previously made to take effect orbecome effective. After the predetermined or predefined duration, theprocessor 102 can compute an average value of luminosity over each newacquired the image. This processor 102 may compare the averageluminosity value to a target range. If the average luminosity value liesin the target range, the processor 102 can determine that the EV (orsensitivity) calibration is successfully finalized and the process canproceed to a next step, e.g., pulsatile signal detection.

If the average luminosity value lies outside the target range, theprocessor 102 can determine whether the value is greater than the targetrange (e.g., when the luminosity is too bright) or the value is smallerthan the target range (e.g., when the luminosity is too dark). Ifluminosity is too bright, the camera calibration process may proceed toa high luminosity update process that can reduce the exposure value by apredetermined factor. On the other hand, if luminosity is too dark, thecamera calibration process may proceed to a low luminosity updateprocess that can increase the exposure value by a predetermined factor.

In some embodiments, the low luminosity update process may be performedas follows. The low luminosity update process may check whether thenumber of the high luminosity update processes previously performed isgreater than a predetermined threshold (e.g., whether too many highluminosity update processes were previously performed to decrease EV).If it is determined that too many high luminosity update processes werepreviously performed to decrease EV, the low luminosity update processmay (1) determine that the user is not stable yet or that the EVoscillates around a target value, (2) issue or state a not-steadywarning, and (3) end or exit. If (1) it is not determined that too manyhigh luminosity update processes were previously performed to decreaseEV and (2) the current value of exposure time is not a maximum value,the low luminosity update process may increase the exposure time valueby a predetermined exposure time update factor (e.g., 1.2) or a computedexposure time update factor responsive to the difference between theactual luminosity and the target luminosity. In some embodiments, thelow luminosity update process may increase the exposure time value andthen increase the sensitivity value. After at least one of thesensitivity or exposure time is updated, the low luminosity updateprocess may submit the new updated gain to the camera. In someembodiments, after submission of a new gain, the low luminosity updateprocess may wait for settings to be effectively taken into accountbefore performing a new update process (either the high luminosityupdate process or the low luminosity update process).

In some embodiments, the high luminosity update process may be performedas follows. The high luminosity update process may check whether thenumber of the low luminosity update processes previously performed isgreater than a predetermined threshold (e.g., whether too many lowluminosity update processes were previously performed to increase EV).If it is determined that too many low luminosity update processes werepreviously performed to increase EV, the high luminosity update processmay (1) determine that the user is not stable yet or that the EVoscillates around a target value, (2) issue or state a not-steadywarning, and (3) abort or end the EV update. If (1) it is not determinedthat too many low luminosity update processes were previously performedto increase EV and (2) the current value of sensitivity is not a minimumvalue, the high luminosity update process may decrease the sensitivityvalue by a predetermined sensitivity update factor (e.g., 1.2) or acomputed sensitivity update factor responsive to the difference betweenthe actual luminosity and the target luminosity. In this manner, thehigh luminosity update process can maintain the sensitivity value andthe exposure time value to be at least the respective minimum values. Insome embodiments, the high luminosity update process may decrease thesensitivity value and then decrease the exposure time value. After atleast one of the sensitivity or exposure time is updated, the highluminosity update process may submit the new updated gain to the camera.In some embodiments, after submission of a new gain, the high luminosityupdate process may wait for settings to be effectively taken intoaccount before performing a new update process (either the highluminosity update process or the low luminosity update process).

According to certain aspects, embodiments in the present disclosurerelate to techniques for detecting a pulsatile signal (e.g., a signalincluding a sequence of pulses) of a user so as to ensure that the userhas its finger well positioned and/or in a good state, in order for PPGsignals to be of good quality. PPG quality is not insured by thehardware alone. A good behavior or condition of the user may bemandatory for the PPG signal to be of good quality. Those cases may becaught by user conditions such as movement of a user, or cold fingerconditions so that these conditions can be considered in processingpixel values captured by a photodetector.

In some embodiments, after the gains are updated to set a PPG signal inan acceptable range, the camera calibration process can proceed to apulsatile signal detection process of detecting a pulsatile signal of auser. The pulsatile signal detection process may calculateautocorrelation of a PPG signal. In some embodiments, the pulsatilesignal detection process may calculate a value of autocorrelation of aPPG signal between consecutive frames and compare the autocorrelationvalue with a predetermined quality threshold. If the autocorrelationvalue is greater than or equal to the quality threshold, the pulsatilesignal detection process may determine that the PPG signal is of goodquality between the consecutive frames. If the autocorrelation value issmaller than the quality threshold, the pulsatile signal detectionprocess may determine that the PPG signal is of poor quality between theconsecutive frames. The pulsatile signal detection process may count thenumber of consecutive frames among which the PPG signal is of goodquality. If the number of consecutive frames of good quality is greaterthan or equal to a predetermined threshold (e.g., 30 frames), thepulsatile signal detection process may determine that a pulsatile signalis detected and the camera calibration process can output theready-to-record state to start the recording. The pulsatile signaldetection process can include the computer system 100 or the processor102 counting the number of cumulative frames among which the PPG signalis of poor quality. If the number of cumulative frames of poor qualityis greater than or equal to a predetermined threshold (e.g., 120frames), the pulsatile signal detection process may (1) determine that apulsatile signal is not detected and (2) abort pulsatile signaldetection.

In some embodiments, the camera calibration process may include theprocessor 102 counting the number of pulsatile signal detectionprocesses (or signal quality assessment processes. If the number ofperformed pulsatile signal detection processes is greater than or equalto a predetermined threshold (e.g., 3), the camera calibration processmay include the processor 102 (1) issuing or raising an error ofcalibration timeout and (2) aborting or ending the calibration.

According to certain aspects, embodiments in the present disclosurerelate to a method including step (a) configuring, according to one ormore predefined data acquisition requirements/rules, one or moreoperational settings of a photodetector of a computing device. Accordingto the method, in step (b), a gain of the photodetector may be adjustedby the computing device. In step (c), after updating the gain, asequence of images may be acquired by the photodetector. In step (d), aphotoplethysmographic (PPG) signal may be generated using the sequenceof images by determining for each image of the sequence of images acorresponding signal value. In step (e), a sequence of local qualityvalues of the PPG signal may be determined. Each local quality value maybe indicative of a local quality of a respective portion of a pluralityof portions of the PPG signal. In step (f), a first predefined number ofconsecutive local quality values exceeding a threshold quality may bedetected among the sequence of local quality values. Upon detecting thefirst predefined number of consecutive local quality values among thesequence of local quality values, a calibration of the photodetector maybe determined to be successful by the computing device.

In some embodiments, a second predefined number of local quality valuesbelow the threshold quality may be detected among the sequence of localquality values, without detecting the first predefined number ofconsecutive local quality values exceeding the threshold quality. Upondetecting the second predefined number of local quality values below thethreshold quality, steps (b)-(e) may be repeated.

In some embodiments, steps (b)-(e) may be repeated for a number ofiterations until detecting the first predefined number of consecutivelocal quality values exceeding a threshold quality, or until the numberof iterations reaches a predefined number of iterations. Upon the numberof iterations reaching the predefined number of iterations withoutdetecting the first predefined number of consecutive local qualityvalues exceeding the threshold quality, an alert message indicative ofcalibration timeout may be provided by the computing device.

In some embodiments, in determining the sequence of local qualityvalues, a sequence of autocorrelation values may be determined. Eachautocorrelation value of the sequence of autocorrelation values may beassociated with a respective signal value of the PPG signal.

In some embodiments, in configuring the one or more operational settingsof the photodetector, a color correction setting of the photodetectormay be disabled. In configuring the one or more operational settings ofthe photodetector, an autocorrection setting of the photodetector may bedisabled. In configuring the one or more operational settings of thephotodetector, a light source associated with the photodetector may beenabled. In configuring the one or more operational settings of thephotodetector, a frame rate of the photodetector may be set to at least30 frames per second.

Embodiments in the present disclosure can have the following advantages.First, some embodiments can provide useful techniques for allowing pixelvalues or a PPG signal to lie in a range where linearity can beapproximated, thereby reducing the likelihood of artifacts due to camerahardware. For example, a process of updating exposure value (EV) can beperformed to set PPG signals to be in an appropriate range of values.

Second, some embodiments can provide useful techniques for configuringsettings of a photodetector (or a camera) to ensure that thephotodetector or camera should be compatible with predeterminedrequirements, for example, that (1) measured numerical values are asclose to underlying physical signal as possible and that (2) a user isable to take measurements in good conditions for PPG signals to be ofgood quality.

Third, some embodiments can provide useful techniques for detecting apulsatile signal of a user so as to ensure that the user has its fingerwell positioned and/or in a good state, in order for PPG signals to beof good quality.

Fourth, some embodiments can provide useful techniques for usingsaturation conditions to perform a transformation of a physical signalto a corresponding digitalized signal (e.g., PPG signal) such that thetransformation is as close to a linear transformation as possible andthe PPG signal can lie in the linear range of the camera.

FIG. 12A shows a flowchart illustrating a method 1200 for calibrating aphotodetector of a computing device for use to estimate blood pressure,according to example embodiments. In brief overview, the method 1200 caninclude configuring, according to one or more predefined dataacquisition requirements or rules, one or more operational settings ofthe photodetector (STEP 1202), and adjusting a gain of the photodetectorby the computing device (STEP 1204). The method 1200 can includeacquiring, by the photodetector after updating the gain, a sequence ofimages (STEP 1206), and generating, using the sequence of images, a PPGsignal by determining for each image of the sequence of images acorresponding signal value (STEP 1208). The method 1200 can includedetermining a sequence of local quality values of the PPG signal, eachlocal quality value indicative of a local quality of a respectiveportion of a plurality of portions of the PPG signal (STEP 1210), anddetermining, by the computing device, a calibration of the photodetectorto be successful upon detecting a first predefined number of consecutivelocal quality values, among the sequence of local quality values,exceeding a threshold quality (STEP 1212).

In further detail, the method 1200 can include the computing device 100configuring, according to one or more predefined data acquisitionrequirements or rules, one or more operational settings of thephotodetector (or camera device) 110 (STEP 1202). In some embodiments,in configuring the one or more operational settings of thephotodetector, a color correction setting of the photodetector may bedisabled. In configuring the one or more operational settings of thephotodetector, an autocorrection setting of the photodetector 110 may bedisabled. In configuring the one or more operational settings of thephotodetector, the computer system 100 can enable or activate the lightsource 108 associated with the photodetector (or camera device) 110. Inconfiguring the one or more operational settings of the photodetector,the computer system 100 or the respective processor 102 can set a framerate of the photodetector 110 to at least 30 frames per second.

In some embodiments, the one or more operational settings of thephotodetector may be initial settings of camera parameters. For example,the gains of the photodetector may be set using previously savedsettings (e.g., exposure time, sensitivity, etc.) and default values forothers. In some embodiments, camera parameters may be set according tothe following rules: (1) the camera is able to take at least apredetermined number of frames per second (e.g., 30 frames per second);(2) output digital pixel intensity values are as close to a lineartransform of an input light intensity as possible; (3) generated PPGsignal values can vary within a range of values (e.g., between 100 and150) that corresponds to an approximately linear portion of the transferfunction of the photodetector while having a wide spread; (4) the flashdevice (or light source) 108 is enabled; (5) no color correction isapplied, or alternatively such transformation should be linear and isapplied in a respective single color channel, e.g., color channel bycolor channel, so that a transformation matrix is diagonal; and/or (6)no autocorrection is applied, or all settings are defined or set by theapplication 114. Color correction and/or autocorrection functionalitiesmay apply nonlinear transformations to incident light intensities orchange exposure value settings during the recording, which can lead todegraded PPG signal quality. Disabling such functionalities avoids ormitigates nonlinear distortions to the PPG signal.

Table 1 below shows an example general camera setting in a cameradevice, according to the above-noted rules.

TABLE 1 A General Camera Setting for Camera Calibration Group Settingkey Value Description General CONTROL_CAPTURE_INTENT TEMPLATE_MANUALThis control helps the 3A algorithms to be set to OFF FLASH_MODE TORCHThis forces the light of the flash to be always on. This way the fingercan be lightened. JPEG_ORIENTATION 0 This forces the camera to be frozenvertically. 3A Algorithms CONTROL_MODE OFF This stops all autocorrection algorithms from camera CONTROL_AE_MODE OFF This stops autoexposure algorithm, forcing the app to use the computed exposure valuessettings and freezing them CONTROL_AE_ANTIBANDING_MODE OFF Remove anyprocessing stabilizing the video. CONTROL_AF_MODE OFF This stops theauto focus algorithm, forcing the zoom level to be stable.CONTROL_AWB_MODE OFF This stops the white level balance updateprocessing, so that relative color ratios stay the same ColorCOLOR_CORRECTION_ABERRATION_MODE OFF This stops the color correctionCorrection aberration mode COLOR_CORRECTION_MODE TRANSFORM_MATRIX Thisforces the color correction algorithm to use a transform matrixCOLOR_CORRECTION_GAINS [1, 3, 3, 3] This gives more strength to thegreen and blue color channels than to the red one since the bloodabsorbs those colors way more than red COLOR_CORRECTION_TRANSFORM [[1,0, 0], [0, 1, 0], [0, 0, 1]] This forces the color transformation matrixto be identity so that it gets as close to the sensor raw data aspossible TONEMAP_MODE CONTRAST_CURVE This forces the mapping between thesensor values and the digitized pixel values to be linear TONEMAP_CURVE[(0, 0), (1, 1)] This forces the mapping between the sensor values andthe digitized pixel values to be linear SHADING_MODE OFF This ensuresthat no lens shading is performed

FIG. 12B shows a flowchart illustrating an example process 1250 forresetting or adjusting operational settings of the camera device 110 or204. The data acquisition module 502 can adjust, or reset, basicsettings, automatic settings and/or color correction settings of thecamera device 110 or 204. Adjusting the basic settings can includeadjusting, or resetting, the sensor frame duration, the sensor exposuretime, the sensor sensitivity and/or the flash mode to correspondingpredefined values. Adjusting the automatic settings can includedisabling one or more automatic settings of the camera device 110 or204, such as auto-white balance (AWB), auto-focus (AF) and/orauto-exposure (AE). Adjusting the color correction settings can includesetting the color correction mode, the color correction gains and/or thecolor correction transform to corresponding predefined values. Forinstance, the data acquisition module 502 can set the color correctionmode to color correction transform, set the color correction gains to[1, 3, 3, 3], and set the color correction transform to an identitymatrix. The data acquisition module 502 can set a tone map mode, acontrast curve and/or a gamma value to corresponding predefined settingvalues. While most cameras usually apply a non-linear gamma correctionto have a greater range of sensitivity, the data acquisition module 502can select settings parameters that lead linear, rather than non-linear,correction.

The method 1200 can include the computer system 100 adjusting a gain ofthe photodetector (STEP 1204). Initially (or in a first iteration), thecomputing device may set the photodetector gain to previously calibratedgain value, e.g., stored in memory 104. If no previously calibrated gainvalue is available, the computer system 100 may set the gain to adefault gain value of the photodetector 110. In adjusting thephotodetector gain, the computer system 100 can adjust an exposure timeand/or a sensitivity of the photodetector. In subsequent iterations,e.g., after determining a sequence of local quality values of a PPGsignal (STEP 1210), the computer system 100 may adjust the gain in a wayto enforce the generated PPG signal values to be within a predefinedrange (e.g., 100 to 150 or other range around 128 in the overall rangeof (0, 255)). For example, the computer system 100 may adjust the gainbased on the PPG signal values by increasing an exposure time and/orsensitivity (e.g., ISO value) or decreasing an exposure time and/orsensitivity. A process of adjusting or calibrating the photodetectorgain, or calibrating the photodetector exposure time and/or sensitivity,according to some embodiments is described in further detail below withreference to FIG. 15 . In other words, in adjusting the gain at STEP1204, the computer system 100 can execute the method 1500 describedbelow in relation to FIG. 15 .

The method 1200 can include the photodetector acquiring, after updatingthe gain, a sequence of images (STEP 1206). The computer system 100 canacquire a sequence of RGB image frames at a preset frame rate (e.g., 25or 30 frames per second). The computer system 100 can use one colorframe or channel (e.g., G or R) of each acquired image frame to generatethe PPG signal. For example, referring to FIG. 7 , the computing devicemay extract a color frame (e.g., a G frame or R frame) 702 from eachacquired RGB image frame and use the sequence of color frames togenerate the PPG signal.

The method 1200 can include the computing device generating, using thesequence of images, a PPG signal by determining for each image of thesequence of images a corresponding signal value (STEP 1208). Asdescribed above in relation to FIG. 7 , the computer system 100 maydownsample each color frame and identify an image sub-block 706 withinthe downsampled color frame 704. The computer system 100 can averagepixel values of each sub-block 606 of the downsampled color frame 704 togenerate a corresponding PPG signal value.

The method 1200 can include the computing device determining a sequenceof local quality values of the PPG signal (STEP 1210). Each localquality value may be indicative of a local quality of a respectiveportion of a plurality of portions of the PPG signal. In someembodiments, in determining the sequence of local quality values, thecomputer system 100 can determine a sequence of autocorrelation values.Each autocorrelation value of the sequence of autocorrelation values maybe associated with a corresponding signal value of the PPG signal. Asdiscussed in further detail below with regard to FIG. 30 , the computersystem 100 can associate a given PPG signal value (e.g., last generatedPPG signal value) or a corresponding image frame (e.g., last acquiredimage frame) with a corresponding signal window of a predefined size ofthe PPG signal. For example, the signal window can be formed of nconsecutive PPG signal values, where n is an integer, and may end at thegiven PPG signal value PPG (e.g., the last acquired PPG signal value).The computer system 100 can compute local autocorrelation values of thePPG signal by incrementally sliding the signal window and computing,after each incremental slide, an autocorrelation value using the signalwindow and another overlapping window of the PPG signal. The computersystem 100 can then determine the maximum autocorrelation value amongthe computed local autocorrelation values. The maximum localautocorrelation value can be viewed as, or can represent, a localquality value associated with the given PPG signal value or thecorresponding image frame. Accordingly, a separate local quality valuecan be computed or determined for each acquired image frame or thecorresponding PPG signal value.

The method 1200 can include the computer system 100 determining acalibration of the photodetector to be successful upon detecting a firstpredefined number of consecutive local quality values, among thesequence of local quality values, exceeding a threshold quality (STEP1212). The computer system 100 can compare each local quality value (ormaximum local autocorrelation value) associated with a correspondingimage frame or corresponding PPG signal value to apredetermined/predefined quality threshold (e.g., 0.85). If the computersystem 100 detects, based on the comparisons, that a first predefinednumber of consecutive local quality values are all greater than or equalto the predetermined/predefined quality threshold, it may determine thatthe generated PPG signal is pulsatile and that the calibration of thephotodetector 110 is successful. An example method of determining that aPPG signal is pulsatile according to some embodiments is described inmore detail below with reference to FIG. 14A. In some embodiments, thecomputing device can maintain a counter of consecutive frames withcorresponding local quality values exceeding predetermined/predefinedquality threshold. If the number of consecutive frames of good qualityreaches or exceeds the first predefined number (e.g., 30 frames), thecomputer system 100 can determine that the calibration of thephotodetector 110 is successful (e.g., PPG signal is pulsatile), and caninitiate a recording of a second PPG signal for use to measure bloodpressure.

In some embodiments, a second predefined number of local quality valuesbelow the threshold quality may be detected among the sequence of localquality values, without detecting the first predefined number ofconsecutive local quality values exceeding the threshold quality. Upondetecting the second predefined number of local quality values below thethreshold quality, steps (b)-(e) may be repeated. For example, thecomputer system 100 or the processor 102 may count the number ofcumulative frames among which the PPG signal is of poor quality. If thenumber of cumulative frames of poor quality is greater than or equal toa predetermined threshold (e.g., 120 frames), it may be determined thata calibration of the photodetector is not successful (or a pulsatilesignal of the user is not detected), and the method may proceed to STEP1204 so that the computing device can adjust the gain such that valuesof the PPG signal are in a middle range of values. A process ofdetermining a calibration of the photodetector to be successful or notaccording to some embodiments will be described in more detail withreference to FIG. 14A.

The computer system 100 or the processor 102 can count all poor qualityframes (not just consecutive poor quality frames) to be compared to thecorresponding predetermined threshold (e.g., 120 frames). However, withregard to good quality frames, the computer system 100 or the processor102 can count or keep track of consecutive good quality frames. Forexample, if a total of 120 poor quality frames (not necessarilyconsecutive) are detected while the total number of detected consecutivegood quality frames is less than 30, the processor 102 can determinethat the calibration of the photodetector is not successful (or a goodquality pulsatile signal of the user is not detected), and the computersystem 100 or the processor 102 may proceed to STEP 1204 so that thecomputer system 100 or the processor 102 can adjust the gain such thatvalues of the PPG signal are in a middle range of values. However, ifthe number of detected consecutive good quality frames is greater thanor equal to 30 and the total number of detected poor quality frames isless than 120, the computer system 100 or the processor 102 candetermine the photodetector calibration process to be successful.

In some embodiments, the computer system 100 may repeat STEPs 1204-1210for a number of iterations until detecting the first predefined numberof consecutive local quality values exceeding a threshold quality, oruntil the number of iterations reaches a predefined number ofiterations. Upon the number of iterations reaching the predefined numberof iterations without detecting the first predefined number ofconsecutive local quality values exceeding the threshold quality, thecomputer system 100 may provide an alert message indicative ofcalibration timeout. The computer system 100 may maintain a counterindicative of the number of iterations that did not lead to a successfulcalibration of the photodetector. For example, if the number ofiterations with no successful calibration is greater than or equal to apredetermined threshold (e.g., 3), the computer system 100 may displayan error of calibration timeout on the display device 112, and thecalibration of the photodetector may be aborted or ended. An exampleimplementation of the method 1200, according to some embodiments, isdescribed in more detail below with reference to FIG. 13 .

FIG. 13 is a flowchart illustrating another method 1300 for calibratingthe photodetector 110, according to some embodiments. The flowchart inFIG. 13 illustrates a method 1300 for calibrating a photodetector (e.g.,camera device 110 in FIG. 1 ), according to inventive concepts of thecurrent disclosure. In brief overview, the method 1300 can includestarting a camera calibration process (STEP 1302), and configuringinitial camera settings (STEP 1304). The method 1300 can includecalibrating gains of camera or photodetector (STEP 1306), and detectinga pulsatile signal of a user to determine whether the camera calibrationprocess is successful or not (STEP 1308). The method 1300 can includedetermining whether the pulsatile signal (of good quality) is detectedor not (STEP 1310), and recording a PPG signal for determining a bloodpressure measurement of the user (STEP 1312).

In further detail, the method 1300 can include a computing device (e.g.,computer system 100) starting a camera calibration process (STEP 1302).

The method 1300 can include the computing device configuring initialcamera settings (STEP 1304). In some embodiments, the computing devicemay configure initial camera settings by at least one of disabling acolor correction setting, disabling an autocorrection setting of thecamera, enabling a light source (e.g., light source 108 in FIG. 1 )associated with the camera, or setting a frame rate of the photodetectorto a predetermined rate, e.g., at least 30 frames per second.

The method 1300 can include the computing device calibrating gains ofcamera or photodetector (STEP 1306). In some embodiments, the computingdevice may calibrate or adjust the gain such that values of a PPG signalare in a middle range of values (e.g., around 128 in the overall rangeof (0, 255)). For example, the computing device may calibrate or adjustthe gain by increasing an exposure time and/or sensitivity (e.g., ISOvalue) or decreasing an exposure time and/or sensitivity. A process ofcalibrating gains of camera according to some embodiments will bedescribed in more detail with reference to FIG. 15 .

The method 1300 can include the computing device detecting a pulsatilesignal of a user to determine whether the camera calibration process issuccessful or not (STEP 1308). In some embodiments, the computing devicemay determine a sequence of local quality values of a PPG signal from asequence of image frames (e.g., frame 602 in FIG. 7 ). The computingdevice may determine whether a pulsatile signal of good quality isdetected or not (or whether the camera calibration process is successfulor not), based on the local quality values of the PPG signal. Forexample, the computing device may determine that a pulsatile signal ofgood quality is detected, based on the number of consecutive frames ofgood quality (e.g., frames with quality values of the PPG signalexceeding a predetermined threshold quality). The computing device maydetermine that a pulsatile signal of good quality is not detected basedon the number of cumulative frames of poor quality (e.g., frames withquality values of the PPG signal smaller than the predeterminedthreshold quality). A process of detecting a pulsatile signal (ordetermining whether the camera calibration process is successful or not)according to some embodiments will be described in more detail withreference to FIG. 14A.

The method 1300 can include the computing device determining whether thepulsatile signal (of good quality) is detected or not (STEP 1310). If itis determined that the pulsatile signal is detected, the method mayproceed to STEP 1316. If it is not determined that the pulsatile signalis detected, the computing device may count the number of executing thepulsatile signal detection (STEP 1312). If the number of executing thepulsatile signal detection is greater than or equal to a predeterminedthreshold (e.g., 3), the camera calibration process may (1) issue orraise an error of calibration timeout (STEP 1314) and (2) abort or endthe calibration. If the number of executing the pulsatile signaldetection is smaller than the predetermined threshold (e.g., 3), thecamera calibration process may proceed to STEP 1306 to adjust orcalibrate the camera gains.

The method 1300 can include the computing device recording a PPG signalfor determining a blood pressure measurement of the user (STEP 1316).

FIG. 14A is a flowchart illustrating a method 1400 for determining asequence of local quality values of a PPG signal, according to someembodiments.

The flowchart in FIG. 14A illustrates a method 1400 of a pulsatilesignal quality determination for determining a sequence of local qualityvalues of a PPG signal for measurement of blood pressure or a pulse of auser, according to inventive concepts of the current disclosure. Themethod 1400 may correspond to STEP 1308 of the method 1300. Thepulsatile signal quality determination process performed by the method1400 can determine whether a pulsatile signal of good quality isdetected or not, in other words, whether the camera calibration processis successful or not. In brief overview, the method 1400 can includestarting the pulsatile signal quality determination process (STEP 1402),and recording one image frame (STEP 1404). The method 1400 can includecomputing and stacking quality values of a PPG signal from image frames(STEP 1406), and determining whether the PPG signal is of good qualityor of poor quality (STEP 1408).

In further detail, the method 1400 can include a computing device (e.g.,computer system 100 in FIG. 1 ) starting a pulsatile signal qualitydetermination process (STEP 1402).

The method 1400 can include the computing device recording one imageframe (STEP 1404). In some embodiments, the computing device may recorda single image frame (e.g., frame 602 in FIG. 7 ) of a user's body part(e.g., finger). The computing device may generate or obtain a resampled(or downsampled) image of the original image frame. For example, aresampled image is a 5×5 small image with macropixels (e.g., image block606 in FIG. 7 ).

The method 1400 can include the computing device computing and stackingquality values of a PPG signal from image frames (STEP 1406). In someembodiments, the computing device may stack numerical values for a givencolor channel (e.g., R or G) to generate a PPG signal. The computingdevice may compute or generate a PPG signal by averaging pixel valuesfor each sub-block of an image frame (e.g., sub-block 606 of color imageframe 602 in FIG. 7 ). The computing device may calculateautocorrelation of the PPG signal. In some embodiments, the computingdevice may calculate a value of autocorrelation of a PPG signal betweenconsecutive frames and compare the autocorrelation value with apredetermined quality threshold (e.g., 0.85; see QUALITY_THRESHOLD inTable 2 below).

The method 1400 can include the computing device determining whether thePPG signal is of good quality or of poor quality (STEP 1408). If theautocorrelation value calculated in STEP 1406 is greater than or equalto the quality threshold, the computing device may determine that thePPG signal is of good quality between the consecutive frames and proceedSTEP 1410 which adds one to a first counter of consecutive good frames.In STEP 1412, if the value of the first counter is greater than or equalto a predetermined threshold (e.g., 30 frames; see PULSE_MIN_GOOD_FRAMESin Table 2 below), the computing device may determine that a pulsatilesignal of good quality is detected (STEP 1414) an proceed to starting arecording of PPG signal (e.g., STEP 1316 in FIG. 13 ).

If the autocorrelation value is smaller than the quality threshold, thecomputing device may determine that the PPG signal is of poor qualitybetween the consecutive frames and proceed STEP 1420 which sets thefirst counter to 0 (zero) and adds one to a second counter of cumulativepoor frames. In STEP 1422, if the value of the second counter is greaterthan or equal to a predetermined threshold (e.g., 120 frames; seePULSE_MAX_POOR_FRAMES in Table 2 below), the computing device maydetermine that a pulsatile signal of poor quality is detected (STEP1424) an proceed to STEP 1312 which determines whether to proceed to thegain calibration (STEP 1306) or proceed to issuing or raising an errorof calibration timeout (STEP 1314).

FIG. 14B shows a flowchart illustrating a method 1450 for assistingfinger placement on the lens during camera calibration, according tosome embodiments. While FIGS. 12A, 13 and 14A do not invoke assessingthe placement of the finger (or other body part) on the photodetectorlens during acquisition of image frames during camera calibration, FIG.14B shows an example process 1450 for detecting presence or absence ofthe finger as well as finger positioning, and displaying proper alertsignals/messages to the user. The computer system 100 can perform thefinger detection as well as the finger positioning as described insubsections D.1 and D.2 below.

Table 2 shows example threshold values used in the pulsatile signalquality determination process.

TABLE 2 Threshold Values for Pulsatile Signal Detection Name ValueDescription QUALITY_THRESHOLD 0.85 Minimal value of local auto-correlation for a single frame to be considered as goodPULSE_MIN_GOOD_FRAMES 30 Minimal number of con- secutive frames toconsider that a pulsatile signal is detected PULSE_MAX_POOR_FRAMES 120Maximal number of frames with poor quality before aborting pulsatilesignal detection

FIG. 15 shows a flowchart illustrating a gain calibration method 1500for updating exposure time and sensitivity of a photodetector or acamera, according to example embodiments. The method 1500 may correspondto STEP 1306 of the method 1300. In some embodiments, the gaincalibration process can update the gains to set a PPG signal in anacceptable range (e.g., a linear response domain) by updating exposurevalues (EVs) of the camera. In some embodiments, the gain calibrationprocess may calibrate or adjust the gain such that values of a PPGsignal are in a middle range of values (e.g., around 128 in the overallrange of (0, 255)) by increasing or decreasing EVs of the camera. Forexample, at the end of the EV update process, PPG signals can be in anappropriate range of values (e.g., around 128 out of 255) if thecalibration is successful, or close to the appropriate range even if thecalibration is not successful. The EVs may include at least one of anexposure time or sensitivity (e.g., ISO value) of the camera.

In brief overview, the method 1500 can include starting the gaincalibration process (STEP 1502), and recording a predetermined durationof signal of image frames (STEP 1504). The method 1500 can includecomputing or obtaining a luminosity value over the image frames (STEP1506), and determining whether the luminosity value is too high or toolow (STEP 1508). The method 1500 can include performing a low luminosityupdate process (or block) (STEP 1512 to STEP 1530), and performing ahigh luminosity update process (or block) (STEP 1542 to STEP 1558).

In further detail, the method 1500 can include a computing device (e.g.,computer system 100 in FIG. 1 ) starting the gain calibration process(STEP 1502).

The method 1500 can include the computing device recording apredetermined duration of signal of image frames (STEP 1504). In someembodiments, the computing device may record a predetermined duration(e.g., 0.5 seconds) of signal of image frames (e.g., image frame 602 inFIG. 7 ) such that an ongoing capture request can have time to be builtand gains can be indeed updated.

The method 1500 can include the computing device computing or obtaininga luminosity value over the image frames (STEP 1506). The computingdevice may compute or obtain an average value of luminosity over one ormore images of the image frames. The computing device may compute orobtain an average value of luminosity over an image frame (e.g., imageframe 602 in FIG. 7 ) by averaging pixel intensities of a correspondingsub-block of the image frame (e.g., sub-block 606 in FIG. 7 ) for eachcolor (e.g., R or G) frame of the image frame. In some embodiments,computing device may compute or obtain an average value of luminosityover an image frame by averaging (1) pixel intensities of acorresponding sub-block of R frame of the image frame and (2) pixelintensities of a corresponding sub-block of G frame of the image frame.

The method 1500 can include the computing device determining whether theluminosity value is too high or too low (STEP 1508). This averageluminosity value may be compared to a target range. Assuming that theluminosity value range is (0, 255), the target range may be around amiddle value, e.g., (128−α, 128+α), α=10%, 20%, or 30% of the middlevalue (13, 26, 39). In some embodiments, the target range may bedetermined based on a linear range of a transfer function. For example,according to the transfer function shown in FIG. 11 , the target rangemay be (100, 150) or another range where the effect of quantization isrelatively low (e.g., less than 1%) compared to the intensity values. Ifthe average luminosity value lies in the target range, the computingdevice may determine that the gain calibration or EV calibration issuccessfully finalized (STEP 1510), end the gain calibration process,and proceed to a next step, e.g., pulse detection (STEP 1308 in FIG. 13).

On the other hand, if the average luminosity value lies outside thetarget range, the computing device may determine that the gaincalibration or EV calibration is not successful, and continue the EVupdate until the average luminosity value lies in the target range. Insome embodiments, the computing device may determine whether the valueis greater than the target range (e.g., when the luminosity is toobright) or the value is smaller than the target range (e.g., when theluminosity is too dark). If luminosity is too dark, the computing devicemay proceed to a low luminosity update process that can increase EVs bya predetermined factor (STEP 1512 to STEP 1530). If luminosity is toobright, the computing device may proceed to a high luminosity updateprocess that can reduce EVs by a predetermined factor (STEP 1542 to STEP1558).

The method 1500 can include the computing device performing the lowluminosity update process (or block) (STEP 1512 to STEP 1530). Thecomputing device may check whether the number of the high luminosityupdate processes previously performed is greater than a predeterminedthreshold (e.g., whether too many high luminosity update processes werepreviously performed to decrease EVs). The computing device maydetermine whether the luminosity value obtained last time (e.g., theluminosity value obtained immediately before obtaining the currentluminosity value) is too low (STEP 1512). Alternatively, the computingdevice may check whether the low luminosity update process was performedlast time (e.g., immediately before the current low luminosity updateprocess was performed).

If it is not determined that the luminosity value obtained last time istoo low, the computing device may determine that the high luminosityupdate process was performed last time, and add one to a high luminosityupdate counter (STEP 1514). If it is determined that the value of thehigh luminosity update counter exceeds a predetermined threshold number,e.g., 3 (STEP 1516), the computing device may determine that too manyhigh luminosity update processes were previously performed to decreaseEVs (which indicates that the user is not stable yet or that the EVoscillates around a target value), and issue a not-steady warning as anerror state and end or exit the gain calibration process (STEP 1520). Ifit is determined that the value of the high luminosity update counter issmaller than the predetermined threshold number, e.g., 3 (STEP 1516),the computing device may (1) determine that there have not been too manyhigh luminosity update processes previously performed to decrease EVsand (2) determine whether the current value of exposure time is amaximum value (STEP 1522). If the current value of exposure time is nota maximum value, the computing device may increase the exposure timevalue by a predetermined exposure time update factor (e.g., 1.2) (STEP1524). On the other hand, if the current value of exposure time is themaximum value, the computing device may determine whether the currentvalue of sensitivity (e.g., ISO value) is a maximum value (STEP 1526).If the current value of sensitivity is not a maximum value, thecomputing device may increase the sensitivity value by a predeterminedsensitivity update factor (e.g., 1.2) (STEP 1528). In other words, if(1) it is determined that there have not been too many high luminosityupdate processes previously performed to decrease EVs, (2) the currentvalue of exposure time is the maximum value, and (3) the current valueof sensitivity is not a maximum value, the computing device may increasethe sensitivity value by a predetermined sensitivity update factor(e.g., 1.2). In this manner, the low luminosity update process canincrease the exposure time while keeping the sensitivity value at theminimum value, which means that the low luminosity update process ratherincreases the exposure time then the sensitivity while the highluminosity update process (see the description below) decreases thesensitivity first and then decrease the exposure time if it is notpossible. After at least one of the sensitivity or exposure time isupdated, the low luminosity update process may submit the new updatedgain to the camera (STEP 1530) and proceed to the recording of new imageframes (STEP 1504). In some embodiments, after submission of a new gain,the low luminosity update process may wait for settings to beeffectively taken into account before performing a new update process(either the high luminosity update process or the low luminosity updateprocess). If the current value of sensitivity is a maximum value, thecomputing device may (1) issue or raise an error of calibration timeout(STEP 1314) and (2) abort or end the gain calibration.

The method 1500 can include the computing device performing a highluminosity update process (or block) (STEP 1542 to STEP 1558). Thecomputing device may check whether the number of the low luminosityupdate processes previously performed is greater than a predeterminedthreshold (e.g., whether too many low luminosity update processes werepreviously performed to increase EVs). The computing device maydetermine whether the luminosity value obtained last time (e.g., theluminosity value obtained immediately before obtaining the currentluminosity value) is too high (STEP 1542). Alternatively, the computingdevice may check whether the high luminosity update process wasperformed last time (e.g., immediately before the current highluminosity update process was performed).

If it is not determined that the luminosity value obtained last time istoo high, the computing device may determine that the low luminosityupdate process was performed last time, and add one to a low luminosityupdate counter (STEP 1544). If it is determined that the value of thelow luminosity update counter exceeds a predetermined threshold number,e.g., 3 (STEP 1546), the computing device may determine that too manylow luminosity update processes were previously performed to increaseEVs (which indicates that the user is not stable yet or that the EVoscillates around a target value), and issue a not-steady warning as anerror state and end or exit the gain calibration process (STEP 1550). Ifit is determined that the value of the low luminosity update counter issmaller than the predetermined threshold number, e.g., 3 (STEP 1546),the computing device may (1) determine that there have not been too manylow luminosity update processes previously performed to increase EVs and(2) determine whether the current value of sensitivity (e.g., ISO value)is a minimum value (STEP 1552). If the current value of sensitivity isnot a minimum value, the computing device may decrease the sensitivityvalue by a predetermined sensitivity update factor (e.g., 1.2) (STEP1554). On the other hand, if the current value of sensitivity is theminimum value, the computing device may determine whether the currentvalue of exposure time is a minimum value (STEP 1556). If the currentvalue of exposure time is not a minimum value, the computing device maydecrease the exposure time value by a predetermined exposure time updatefactor (e.g., 1.2) (STEP 1558). In other words, if (1) it is determinedthat there have not been too many low luminosity update processespreviously performed to increase EVs, (2) the current value ofsensitivity is the minimum value, and (3) the current value of exposuretime is not a minimum value, the computing device may decrease theexposure time value by a predetermined exposure time update factor(e.g., 1.2). In this manner, the high luminosity update processdecreases the sensitivity value first and then decrease the exposuretime if it is not possible. After at least one of the sensitivity orexposure time is updated, the high luminosity update process may submitthe new updated gain to the camera (STEP 1560) and proceed to therecording of new image frames (STEP 1504). In some embodiments, aftersubmission of a new gain, the high luminosity update process may waitfor settings to be effectively taken into account before performing anew update process (either the high luminosity update process or the lowluminosity update process). If the current value of exposure time is aminimum value, the computing device may (1) issue or raise an error ofcalibration timeout (STEP 1314) and (2) abort or end the gaincalibration.

D. Improving Accuracy of Blood Pressure Measurements by EnhancingAcquisition of Transdermal Optical Data

The accuracy of the blood pressure measurements or heart ratemeasurements output by method 400 depends on the quality of thegenerated PPG signal, which in turn depends on the acquired transdermaloptical data or sequence of image frames acquired by the camera device110. Specifically, the less noisy the image frames acquired by thecamera device 110 and the more accurately they reflect the lightintensity reflected from the user finger (or other body part), the moreaccurately the corresponding PPG signal mimics the pulse waveform of theuser. A PPG signal of good quality is more likely to contain moreinformation about blood pressure and other physiological data than oneof poor quality. Feeding a “cleaner” PPG signal (e.g., with respect tohow accurately it mimics the pulse waveform of the user) to the OBPMmodule 510 leads to more accurate blood pressure and/or pulse (or heartrate) measurements output by the OBPM module 510.

During the acquisition of optical signals (or sequence of image frames)through the camera device 110, there are several factors that cancompromise the quality of the recorded image frames or optical data.Among these, there are several that involve the interaction of theuser's finger (or other body part) with the camera' optical waveguide orthe corresponding transparent glass. Other factors relate to noise ordistortions originating at the camera device 110. For instance, manycamera devices apply some image processing techniques to captured imagesor image frames, for example, to enhance contrast, brightness, sharpnessor other visual characteristics. The image processing techniques appliedby the camera device 110 can be nonlinear in nature, therefore,introducing a distortion in the in the corresponding PPG signal.

To address these problems, the computer system 100 or the application114 can guide the user to position the finger (or other body part)properly on the camera lens or transparent glass, to apply appropriatepressure, and to maintain a stable position during recording. Thecomputer system 100 or the application 114 can guide the user before thestart of data acquisition by the camera device 110 and/or after thestart of data acquisition. For instance, the computer system 100 or thesignal quality assessment module 508 can assess the quality of colorsignals generated based on acquired image frames, and cause the outputmodule 512 to present feedback to the user, if needed. As to thedistortions originating at the camera device 110, the data acquisitionmodule 502 can adjust one or more settings of the camera device 110.

To correct finger (or other body part) location, the computer system 100or the application 114 can detect pre-recording and/or intra recordingfinger location relative to the lens and provide user feedback andguidance via the UI(s). Also, the computer system 100 or the application114 can detect pre-recording and/or intra recording finger pressureagainst the optical waveguide leading to the lens, the correspondingtransparent glass or the lens, and provide user feedback and guidancevia the UI. In some implementations, to detect the location or pressureof the user finger (or other body part), the computer system 100 orsignal quality assessment module 508 can assess color signals generatedbased on acquired image frames based on one or more correspondingcriteria or rules.

Different finger sizes, as well as variation of fingerplacement/orientation over the camera lens usually results in differentillumination characteristics, including partial occlusion of the imagefield and variation in the location of the optimal region or block ofthe image field for constructing or generating the PPG signal to beinput to the OBPM module 510. In addition, during the acquisition ofimage frames, the user's finger (or other body part) can move causingthe finger position, orientation or pressure to change, and thereforedegrading the quality of the PPG signal. The computer system 100 or thesignal quality assessment module 508 can iteratively assess the imagedata or the corresponding PPG signal, for example, before and during theacquisition of the image frames.

D.1. Finger Detection

The computer system 100 can capture, record, or obtain PPG signals usingat least one camera device 110 and at least one light source 108 for thecamera device 110 to capture the light intensity reflected from the userfinger, among other body parts. The computer system 100 executing theapplication 114 can use the PPG signals to determine or estimate theamount of blood present in the finger of the user. The computer system100 (or other devices processing the PPG signals) can obtain good PPGsignals for accurate measurement by having the user place their fingerwell on a device capturing the PPG signals, such as the camera device110. By placing the finger well on the device, the computer system 100can obtain, output, or generate good results to accurately measure atleast the amount of blood present in the finger of the user, among otherhealth features of the user. For example, the computer system 100 canprocess a sequence of images to make sure that the user places theirfinger well on the camera device 110. The computer system 100 maydetermine that the user is placing their finger well on the cameradevice 110. In some cases, the computer system 100 may determine thatthe user is not placing their finger well (e.g., displacement from thecenter, pressure applied, or lighting reflected from a finger) or thattheir finger is not on the camera device 110. To detect that a finger ofthe user is well placed or present on the capturing device (e.g., cameradevice 110), the application 114 executing on the computer system 100can use an algorithm to distinguish or discriminate frames with orwithout a finger.

The application 114 can use Laplacian or other derivativeoperators/imaging techniques for finger detection, such as to detecttexture in the image frames. Laplacian can be a measure of variations ofmultidimensional signal, calculated by using second derivatives over xand y axes. Using Laplacian, the application 114 can highlight regionsof rapid intensity changes (e.g., the intensity of the color in an imageframe), which can be used for edge detection for the image. For example,the application 114 can consider image frames with little textures asbeing a finger and image frames with an abundance of textures as notbeing a finger, in some cases. The use of the Laplacian may notdistinguish between texture of a finger or other objects having no orlittle texture. Such distinction can be made by other processesdescribed in this disclosure, such as the signal quality assessmentmethod of FIG. 29 . The application 114 can use Laplacian to measure theuniformity and texture of each downsampled image using local variationvalues at each pixel position. Specifically, the application 114 canapply the Laplacian equation for individual pixels of image frames todetermine characteristics of the image frame, such as whether there arevarious different textures or uniformed textures within an image todetect the presence of a finger on the camera device 110, for example.

By using Laplacian on downsampled images (e.g., 5×5 images or frames),the application 114 can efficiently detect whether a frame displays afinger or if no finger is present within the respective frames. Forexample, the computer system 100 executing the application 114 cancompute the Laplacian of a sequence of images (e.g., a sequence ofdownsampled images) to determine how much texture is captured inindividual image frames. If a finger is present (e.g., against thecamera device 110, photodetector, or photodetector lens when capturingthe image), the computer system 100 can compute one or more valuesindicating that the image has little texture. Images frames with littletexture can include uniform images. Specifically, the computer system100 can compute the Laplacian of images to determine the uniformity ofindividual images represented by at least one value.

For computation, the computer system 100 can provide one or more imageframes or a sequence of downsampled images as input(s) for the Laplacianoperator. The input can be denoted as X, at least for this section, andhas shape (d_(x), d_(y), 3). The d_(x) and d_(y) can represent theheight and width of an image frame used as the input, respectively. Insome cases, the d_(x) and d_(y) of the shape can represent the x-axispixels and y-axis pixels of an image frame, respectively. The constant 3can represent the number of color channels (e.g., R, G, or B) for theshape. For example, the computer system 100 can provide one or moresequences of downsampled images as input for the Laplacian operator,where each sequence can include one of the color channels. The computersystem 100 can compute the Laplacian of individual pixels for therespective downsampled images having a respective color channel (e.g., asequence of G downsampled images or a sequence of R downsampled images).

The computer system 100 can output or generate a transformed imageincluding the Laplacian of the image for each color channel. TheLaplacian can refer to a normalized (e.g., denoted as L or raw averageLaplacian of an image frame. The Laplacian can be denoted as L. Forexample, the computer system 100 can output a first Laplacian of asequence of downsampled images in the G color channel, a secondLaplacian of a sequence of downsampled images in the R color channel,and a third Laplacian of a sequence of downsampled images in the B colorchannel. The Laplacian output by the computer system 100 can include ashape of (d_(x)−2, d_(y)−2, 3), e.g., height and width of two less thanthe input height and width. Each value L[x,y,c] representing each imageframe of the sequence of downsampled images can include the localLaplacian of X (or of the image) on color channel c around pixel (x+1,y+1).

Referring to FIG. 16 , a flowchart illustrating a method 1600 ofdetecting a finger is shown, according to inventive concepts of thisdisclosure. The method 1500 can include acquiring a sequence of images(STEP 1602). The method 1600 can include generating a sequence ofdownsampled images of the sequence of images (STEP 1604). The method1600 can include determining a respective local variation value for eachpixel of various pixels in the downsampled image (STEP 1606). The method1600 can include determining a respective metric based on the localvariation values of each pixel of the various pixels of the downsampledimage (STEP 1608). The method 1600 can include classifying eachdownsampled image as usable or unusable for measuring blood pressureusing the respective metric (STEP 1610). The method 1600 can beperformed by the computer system 100 or the corresponding processor 102,for example, upon executing the application 114. The method 1600 can beperformed by other components (e.g., memory 104, BUS 106, light source108, camera device 110, or display device 112) of the computer system100, in conjunction with FIG. 1 . The method 1600 can be performed by aremote device or a remote server. In some implementations, the method1600 can be performed by a computer system 100 and a remote server. Forexample, the computer system 100 can establish a communication channelwith the remote server to forward data to or receive processed data fromthe remote server. The computer system 100 can acquire the sequence ofimages and pass the sequence of images to a remote server. Responsive toreceiving the sequence of images, the remote server can perform STEPS1004-1010. After classifying the downsampled image, the remote servercan transmit the processed data or the classification of the downsampledimage to the computer system 100. Accordingly, each STEP of method 1500can be performed by a combination of the computer system 100 and theremote server. The method 1500 can be performed by other componentsdiscussed herein and perform features and functionalities in conjunctionwith FIGS. 1-10 , for example.

Still referring to FIG. 16 , in further detail, a photodetector of thecomputer system 100 can acquire a sequence of images (STEP 1602). Thephotodetector can be referred to as a photosensor, which can include orbe the light source 108. The sequence of images can representtransdermal optical data of a subject. A user can actuate theapplication 114 on the computer system 100, and apply their fingertip orother body parts to the camera 110 for example, as illustrated in FIGS.2A, 2B, and 3A. Upon initiation, the application 114 can cause thecomputer system 100 to display a user interface (UI) on the displaydevice 112. The UI can provide instructions to the user to apply theirfingertip against the camera 110 or a corresponding transparent glassfor a given period of time, e.g., 20 seconds. The UI can provide anoption, e.g., upon user selection, to present further instructions,images, or a demo explaining how the fingertip (or other body parts) isto be applied to the camera 110.

The application 114 or the data acquisition module 502 can automaticallytrigger the camera 110 to acquire a sequence of image frames. In someimplementations, the application 114 or the data acquisition module 502can cause the computer system 100 to display an interactive item as partof the UI. The application 114 or the data acquisition module 502 canautomatically trigger the camera device 110 to acquire the sequence ofimages, upon the user interacting with the interactive item. Triggeringthe camera device 110 can include actuating the flash device 206 to emitartificial light either continuously or periodically according to agiven frequency.

The camera device 110 can acquire the sequence of image frames while theuser's fingertip (or other body parts) is applied against the opticalwaveguide of the camera or the corresponding transparent glass.Specifically, the RGB sensor 312 can acquire the sequence of imageframes according to a specific or predefined frame rate based on thelight intensity I_(r) reflected from the user's finger. The frame ratecan be equal to 25 frames/second (fps), 30 fps, 50 fps or other framerate supported by the camera device 110. The data acquisition module 502can receive image data corresponding to image frames from the cameradevice 110. In some implementations, the data acquisition module 502 canreceive image data for each frame immediately after the image frame iscaptured or recorded by the RGB sensor 312 or the camera device 110.

In some implementations, the computer system 100 can include a pluralityof cameras 110. The computer system 100 can automatically select one ofthe cameras 110 to acquire the sequence of image frames, and display(e.g., via a user interface of the application 114) a notification tothe user of the computer system 100 of the selected camera 110. The usercan place his/her finger against the selected camera. The selection canbe based on the features or characteristics of the various cameras. Insome implementations, the user can select the camera to be used toacquire the sequence of image frames. The computer system 100 canreceive or obtain the user selection via an input device, e.g., akeyboard, a mouse, or a touch screen, among others, of the computersystem 100. The computer system 100 may confirm the user selection via,for example, the display device 112. The computer application 114 mayinclude different settings for distinct cameras 110. The computer system100 can maintain camera ID/phone model matrix in the memory 104. Besidesthe camera ID, the computer system 100 can store the preferred camerasettings in the matrix, such as ISO, exposure time, tone map, RGB gainor a combination thereof, among other settings.

The computer system 100 can generate a sequence of downsampled images bydownsampling each image of the sequence of images (STEP 1604). Thesequence of images can include an image region or block associated withat least one color (e.g., red (R), green (G), or blue (B)) frame 702.The data acquisition module 502 of the application 114 executing on thecomputer system 100 can decompose each image into a respective color(e.g., R, G, or B) frame. The computer system 100 can generate thesequence of downsampled images responsive to or immediately afteracquiring or obtaining the images of the subject. In some cases, thecomputer system 100 can delegate the generation of the sequence ofdownsampled images to a remote server or a remote computing device. Forexample, the computer system 100 can acquire and transmit the sequenceof images to a remote server to process the sequence of images.Processing the sequence of images can include downsampling the sequenceof images to generate a sequence of downsampled images. The sequence ofdownsampled images can be further analyzed or processed by the remoteserver or transmitted to the computer system 100.

The computer system 100 can downsample each image of the sequence ofimages. The computer system 100 can perform the downsampling process inconjunction with FIG. 4 and FIG. 7 or execute the features orfunctionalities as described in FIG. 4 and FIG. 7 . For example, theprocessing module 504 of the application 114 executing on the computersystem 100 can identify an image region or block associated with a color(e.g. G or R) frame 702. The image can be from one of the sequence ofimages. The processing module 504 can downsample each color (e.g. G orR) frame 702 of the sequence of image frames to smaller image block 704.The processing module 504 can downsample the color (e.g. G or R) frame702 to a 5×5 image (or data) block, a 9×9 image (or data) block, or adata block 704 of a different size. In downsampling the color frame 702,the processing module 504 can apply low-pass filtering or averaging tothe pixel values of the color frame 702. The processing module 504 candownsample the color frame 702 using known downsampling techniques.

The processing module 504 can identify a sub-block 706 of the imageblock 704 for use to generate a color signal. The processing module 504can identify a central zone or sub-block of the image block 704 assub-block 706. Depending on the size and the placement of the userfinger against the camera, the color intensities of a central zone ofthe image block 604 (or the color frame 702) are more likely to reflectthe pulsatile nature of arterial blood in the finger than outer regionsof the image block 704 (or the color frame 702). In some cases, the userfinger may not be placed right at the center of the camera device 110,but rather shifted towards one direction over another. In such cases, atleast one outer region of the image frame may not adequately reflect thelight (or optical signal reflected from the user finger. In someimplementations, the identified sub-block 706 can be a 3×3 image (ordata) block, a 5×5 data block, or a data block of other sizes. Bydownsampling individual images of the sequence of images, computationalresources used to process the images, as well as resource consumptionfor storing the images that are not downsampled, can be reduced. Forexample, by downsampling images with thousands of pixels (e.g., 320×240,720×576, 1024×768, among other image sizes) to a 3×3 image, the computersystem 100 can significantly reduce computational resources used toprocess the downsampled images and resource consumption to store thedownsampled images.

The computer system 100 can determine a respective local variation valuefor individual pixels in each downsampled image of the sequence ofdownsampled images (STEP 1606). The local variation value can include orbe referred to as a Laplacian value. The computer system 100 can computethe local variation value for each pixel using a Laplacian operator or aLaplacian equation. The computer system 100 can determine the respectivelocal variation value based on a color value of the pixel and one ormore color values of adjacent pixels. In some cases, the computer system100 can determine the respective local variation value based on a colorintensity of the pixel and one or more color intensities of adjacentpixels. The adjacent pixels can be pixels above, below, or to the sidesof the pixel being computed for the local variation value. In this case,the adjacent pixels can be contiguous (e.g., forming a cross (“+”)) withthe pixel to compute the local variation value. The computer system 100can compute the local variable value for the inner pixels of each imageframe from the sequence of downsampled images. For example, in a 5×5image, the local variation value can be computed for the inner 3×3pixels of the image.

For example, the computer system 100 (or a remote server) executing theapplication 114 can use Laplacian on individual pixels of each imageframe of a sequence of downsampled images. The Laplacian can bedescribed using the following formula/equation (e.g., sometimes referredto as a Laplacian formula or formula (7)):

$\begin{matrix}{{\Delta{X( {x,y} )}} = {{\overset{arrow}{\nabla} \cdot {\overset{arrow}{\nabla}{X( {x,y} )}}} = {{\frac{\partial^{2}X}{\partial x^{2}}( {x,y} )} + {\frac{\partial^{2}X}{\partial y^{2}}{( {x,y} ).}}}}} & (7)\end{matrix}$

The denoted x and y can represent the pixel location that the computersystem 100 is computing the local variation value for. The denoted{right arrow over (∇)}·{right arrow over (∇)}X (x, y) and

${\frac{\partial^{2}X}{\partial x^{2}}( {x,y} )} + {\frac{\partial^{2}X}{\partial y^{2}}( {x,y} )}$

can represent the double derivative of the X(x, y) function. In otherwords, the Laplacian can be described as or calculated by taking doublederivative each pixel of the image X having shape (dx, dy, C). Theformula (7) can be used for a continuous signal. However, since theimage frames or downsampled images are discrete (e.g., each pixel havinga discrete value representing the color value or color intensity for arespective color channel), the computer system 100 can use an updatedLaplacian for determining the local variation value of individualpixels. The updated formula can be described in formula (8) (e.g.,sometimes referred to as the discrete Laplacian formula):

$\begin{matrix}{{\Delta{X\lbrack {x,y} \rbrack}} = {{{\frac{\partial^{2}X}{\partial x^{2}}\lbrack {x,y} \rbrack} + {\frac{\partial^{2}X}{\partial y^{2}}\lbrack {x,y} \rbrack}} = {{X\lbrack {{x + 1},y} \rbrack} - {2{X\lbrack {x,y} \rbrack}} + {X\lbrack {{x - 1},y} \rbrack} + {X\lbrack {x,{y + 1}} \rbrack} - {2{X\lbrack {x,y} \rbrack}} + {{X\lbrack {x,{y - 1}} \rbrack}.}}}} & (8)\end{matrix}$

Specifically

$\frac{\partial^{2}X}{\partial x^{2}}\lbrack {x,y} \rbrack$

can be equated with X[x+1,y]−2X[x,y]+X[x−1,y] and

$\frac{\partial^{2}X}{\partial y^{2}}\lbrack {x,y} \rbrack$

can be equated with X[x,y+1]−2X[x,y]+X[x,y−1] Using the discreteLaplacian formula (e.g., formula (7)), the outer pixels can be cropped(e.g., local variation values may not be calculated for the outerpixels), as denoted in equation x+1, x−1, y+1, and y−1, since thecomputer system 100 may not extend to a null value or a value outsidethe image frame to calculate the local variation values.

The computer system 100 can use the formula (8) to calculate arespective local variation value for individual pixels of an imageframe. The computer system 100 can use formula (8) for interior pixels.For example, in a 5×5 image, the computer system 100 can calculate oneor more local variation values for each pixel in the 3×3 inner region ofthe 5×5 image. In some embodiments, with x and y representing the pixellocation for calculating the local variation value and the X[x, y] canrepresent the color value (or other values) corresponding to the pixellocation. The computer system 100 can use the discrete Laplacian formulato determine the local variation value of each interior pixel. Forexample, X[x+1, y] can correspond to a color value of the pixel atposition (x+1, y), X[x−1, y] can correspond to a color value of thepixel at position ([x−1, y), X[x, y+1] can correspond to a color valueof the pixel at position (x, y+1), and X[x, y−1] can correspond to acolor value of the pixel at position (x, y−1). Each of theaforementioned functions can represent the color value of the respectivepixel. To simplify, the discrete Laplacian formula can be calculated bytaking the sum of color values adjacent to a respective pixel (e.g.,pixel at [x, y] pixel location), such as the sum of four color values ofthe adjacent pixels, and subtract the sum by 4 times the color values at[x, y] pixel location. The computer system 100 can perform similarprocedures for other pixels. Accordingly, the computer system 100 candetermine the local variation value for individual pixels for each imageframe, where the local variation value can represent the magnitude ofsimilarity, uniformity, or differences between a pixel (e.g., a firstpixel) and the adjacent pixels (e.g., pixels contiguous to the firstpixel).

The computer system 100 can determine a metric based on or using therespective local variation values of each of the pixels of thedownsampled image (STEP 1508). The metric can be referred to as auniformity metric or a texture metric, and can be indicative of theuniformity of (or a measure of texture in) individual downsampledimages. The computer system 100 can aggregate the local variation valuesof the respective pixels of each downsampled image to determine a singlescalar value (e.g., the metric). To aggregate, the computer system 100can compute an average or mean of the local variation values. In somecases, the computer system 100 can determine the metric by determiningthe average Laplacian value using the respective Laplacian value foreach pixel of individual downsampled images. Formula (9) can be used todetermine the Laplacian of the image:

$\begin{matrix}{L = {\frac{1}{( {d_{x} - 2} ) \cdot ( {d_{y} - 2} )}{\sum}_{x = 1}^{d_{x} - 2}{\sum}_{y = 1}^{d_{y} - 2}\Delta{{X\lbrack {x,y} \rbrack}.}}} & (9)\end{matrix}$

To compute L, the computer system 100 can compute a sum of all localvariation values, as represented by Σ_(x-1) ^(d) ^(x) ⁻²Σ_(y-1) ^(d)^(y) ⁻²ΔX[x,y]. If processing a 5×5 image block, the computer system 100or the processor 102 can sum up Laplacian or local variation values for(d_(x)−2)×(d_(y)−2) or 3×3 pixels. In this example, the computer system100 can compute a sum of local variation values at pixel locations [1,1], [1, 2], [1, 3], [2, 1], [2, 2], [2, 3], [3, 1], [3, 2], and [3, 3],where the indexing of pixel locations is from 0 to 4. After summing thenine values, the computing device can divide the sum by the total numberof pixels the local variation values were computed from. In this case,the product of (d_(x)−2) and (d_(y)−2) is 9. The denominator can be(d_(x)−2) and (d_(y)−2), due to cropping the outer pixels of each imageframe when using the discrete Laplacian formula. Accordingly, thecomputer system 100 can determine L or the metric based on the localvariation values.

In some implementations, the computer system 100 can normalize themetric (or the Laplacian metric) into a normalized metric (or anormalized Laplacian metric). By normalizing the metric, the computersystem 100 can account for the fact that high-valued images are likelyto vary more than low-valued images. The computer system 100 can utilizethe normalization formula (e.g., formula (10)) to compute the normalizedmetric denoted as L:

$\begin{matrix}{\overset{\sim}{L} = {\frac{L}{\overset{\_}{X}}.}} & (10)\end{matrix}$

The variable X can represent the mean of the downsampled image or themean of central block (e.g., 3×3) of the downsampled image. For example,as a first step to calculate the standard deviation, the computer system100 can compute the mean or the average of the local variation values,which is L. Second, the computer system 100 can calculate the mean ofthe downsampled image. Accordingly, using L and X, the computer system100 can determine the normalized metric L.

The computer system 100 can classify each downsampled image as usable orunusable for measuring blood pressure using the respective metric (STEP1510). For example, the computer system 100 can compare a metric of therespective downsampled image to a threshold. The threshold can includeor be referred to as a quality threshold. By satisfying the threshold,the computer system 100 can classify the downsampled image as usable formeasuring blood pressure, for example. Otherwise, if the metric does notsatisfy the threshold, the computer system 100 can classify thedownsampled image as unusable. In some implementations, the computersystem 100 can classify each downsampled image as usable or unusable formeasuring blood pressure using a respective normalized metric. In suchimplementations, the computer system 100 can compare the normalizedmetric of the respective downsampled image to a second threshold (e.g.,a second quality threshold for normalized metric).

The computer system 100 can determine the threshold using a machinelearning model trained using various image frames or at least onesequence of images. The machine learning model can operate on thecomputer system 100 or a remote server, where the computer system 100can forward the image frames for processing/classification by the remoteserver. For example, the computer system 100 can feed or input imagedata of image frames captured by the camera device 110 or other camerasfrom other devices into a machine learning model. For each image, thecomputer system 100 can provide an indication of whether a user finger(e.g., or other body parts) is against the camera (e.g., should orshould not be classified as detecting the finger). In some cases, thecomputer system 100 can provide an indication of whether the user fingercovers the entire camera, cover a portion of the camera, or away fromthe camera. The machine learning model can process the image data todetermine the metric (e.g., averaged Laplacian value or normalizedLaplacian value) of one or more respective images. In some cases, thecomputer system 100 can also provide the machine learning model with themetrics associated with the respective image frames. The machinelearning model can determine based on the metrics for input image framescorresponding to existing finger(s) and the metrics of image framescorresponding to no finger one or more thresholds for the metric. Thecomputer system 100 can use a range (e.g., from a minimum value to amaximum value) as the threshold or a single threshold value (e.g.,greater than a minimum value or less than a maximum value) to determineif the user finger is present. In some implementations, the computersystem 100 can determine if the user finger is well placed (e.g.,covering the entirety of the camera) based on the metric compared to thethreshold obtained by the machine learning model. In someimplementations, a threshold can be determined by an administrator ofthe computer system 100 or the application 114.

The computer system 100 can use the metric to determine the texture oruniformity of each image frame. By using Laplacian, a small amount oftraining is sufficient for the machine learning model to determine athreshold for the computer system 100 to accurately detect if the userfinger is present. Thus, the computer system 100 can train a goodthreshold for Laplacian to discriminate or distinguish frames with nofinger from those with a user finger with minimum false negatives. Insome implementations, the computer system 100 can train a differentmachine learning model without Laplacian computation to obtain athreshold and a metric to detect the presence of a user finger.

In some implementations, the computer system 100 can compute a bloodpressure value using various downsampled images classified as usable.For example, the computer system 100 can determine that the respectivemetrics of one or more image frames satisfied a threshold. The computersystem 100 can classify the image frames as usable for measuring bloodpressure. Responsive to the determination, the computer system 100executing the application 114 can initiate or transmit an instruction tothe OBPM module 510 to measure at least one blood pressure based on thecaptured sequence of images (e.g., downsampled images). In some cases,the computer system 100 can instruct the OBPM module 510 to compute orestimate the heart rate or pulse rate of the user based on the sequenceof images.

In some implementations, the computer system 100 can generate an alertresponsive to determine that a predetermined number of downsampledimages are classified as unusable. For example, the computer system 100can determine that the respective metrics of one or more image frames donot satisfy a threshold. In this case, the computer system 100 canclassify the image frames as unusable. Responsive to classifying one ormore image frames as unusable, the computer system 100 executing theapplication 114 can initiate an alert to be presented on the UI todisplay to the user. The alert can indicate that the computer system 100is unable to detect the presence of the user finger. In some cases, thecomputer system 100 can transmit an alert responsive to a time thresholdfor capturing unusable image frames. For example, the computer system100 can be configured to alert the user after 1 second, 2 seconds, 3seconds, 5 seconds, etc. after not receiving one or more usable imageframes. In some implementations, the computer system 100 can beconfigured to transmit an alert after a predetermined number of imageframes classified as unusable, such as 20 frames, 40 frames, 60 frames,etc.

FIG. 17 illustrates examples of individual downsampled frames or images1702-1716 of the sequence of downsampled images. The downsampled imageframes can be generated by the computer system 100, the application 114executing on the computer system 100, or a remote server, such asexecuting STEP 1604 in conjunction with FIG. 16 . In this example, thedownsampled images can include 5×5 images containing at least one colorvalue and one or more color intensities (or pixel intensities). Eachimage can include or correspond to a color (e.g., R or G) frame based onthe configuration of the computer system 100. For example, the computersystem 100 can extract individual colors (e.g., R, G, or B) from asequence of images and output a second sequence of images with one ofthe colors corresponding to a color channel, such as R or G. Each regionor pixel of the image can include a color (e.g., G, R, or B) having arespective color value and color intensity (e.g., averaged colorintensity) of the color. In some cases, the color value can refer to thebrightness of the color (e.g., ranging from 0% to 100%, where a lowerpercentage represents a darker color and a higher percentage representsa brighter or whiter color). The color intensity can refer to thesaturation of the color (e.g., ranging from 0% to 100%, where a lowerpercentage indicates no color, which can be a shade of gray, and ahigher percentage can represent a more intense or vibrate color). Insome implementations, the color intensity may be interchanged with thecolor value for detecting the user finger. In some implementations, thedownsampled images can be generated with different pixel densities, suchas 10×10 images, 25×25 images, among others.

The computer system 100 can use the downsampled image frames of FIG. 17to determine if a finger of the use is on the camera device 110. Forexample, the camera device 110 can capture, or the computer system 100can obtain the image frames. Depending on the size and the placement ofthe user finger against the camera, the color intensities of a centralzone of the image (or the color frame) are more likely to reflect thepulsatile nature of arterial blood in the finger than outer regions ofthe image (e.g., image block 704 or the color frame 702). Further,depending on if the user finger is against or away from the camera, thetexture of each image frame may be more uniform (e.g., having littletexture indicating that the user finger is on the camera) or lessuniform (e.g., having more texture indicating that the user finger isaway from the camera), for example. In some cases, having no texture orcomplete uniformity in color intensities (or color values) may indicatethat the finger is not against the camera. In some cases, having a colorintensity higher than a threshold (e.g., above 90% intensity) canindicate that at least a portion of the user finger is away from or notcovering the camera, such that other lights (e.g., room lighting ornon-reflective lights directed from the light source 108) is captured bythe camera.

Referring to FIG. 17 , the computer system 100 can compute the metricfor each image frame based on the local variation values of individualpixels (e.g., inner 3×3 pixels of the 5×5 pixels). The value associatedwith the respective image frames (e.g., 0.00, 0.42, 0.98, 3.13, and1.58) can represent the metric calculated for the respective imageframes. In this example, with more texture or the greater the Laplacianof the image frame, the computer system 100 can compute a higher valuedmetric. Based on the threshold, the computer system 100 can discriminateor distinguish between image frames with a user finger or image frameswithout a user finger.

For example, referring to the three image frames 1702, 1704 and 1706 ofthe top row, the computer system 100 can compute metrics with a lowvalue, indicating a high uniformity of the pixels or a small amount oftexture. In another example, referring to the image frames 1714 and1716, the computer system 100 can compute metrics with higher values(e.g., 3.13 and 1.58), indicating a low uniformity of the pixels or highamount of texture. If the threshold is set to 0.5, the computer system100 can classify image frames 1702, 1704, 1706, 1708 and 1710 as usable,e.g., all of which are below the threshold. The computer system 100 mayclassify other image frames 1712, 1714 and 1716 as unusable due to therespective metrics surpassing the threshold, for example.

The computer system 100 can determine if the body part (e.g., a finger)of the user is present on the camera lens to measure the PPG of theuser. Having the body part present on the camera is one of variousconditions assessed by the computer system 100 to proceed with measuringthe PPG. The computer system 100 can use a single image frame todetermine, at least the likelihood of, whether the finger is on the lensof the camera or photodetector. The computer system 100 can downsample asequence of image frames captured by the camera to generate a sequenceof downsampled images. Each of the downsampled images can include adimension of 5×5 pixels or other predetermined dimensions.

The computer system 100 can compute the Laplacian of each image frameusing one or more techniques or formulas described above. Based on theLaplacian, the computer system 100 can attribute three normalizedLaplacian values (e.g., denoted as LC) for each color channel denoted asc. For example, a first normalized Laplacian value can correspond to thegreen color channel, a second normalized Laplacian value can correspondto the red color channel, and a third normalized Laplacian value cancorrespond to the blue color channel. The computer system 100 cancompare the normalized Laplacian values to one or more thresholdscorresponding to the respective normalized Laplacian values. The one ormore thresholds can be denoted as θ_(p) in the finger. For example, thecomputer system 100 can compare the normalized Laplacian values to thethreshold to determine if the Laplacian should be considered as a finger(or a body part) of the user. In some implementations, the threshold canbe set to 0.8 (or other threshold value) for at least one of the colorchannels, such as red or green. In some cases, the threshold may not beset for one of the color channels, such as blue.

In some implementations, the computer system 100 can capture anddownsample a sequence of images in a single color channel, such as greenor blue. In this case, the computer system 100 can compute a singlenormalized Laplacian value for each of the sequence of downsampledimages. The single normalized Laplacian value corresponding to therespective color channel. Accordingly, the computer system 100 cancompare the normalized Laplacian value to the respective thresholdcorresponding to the color channel similar to the normalized Laplacianvalue. In some implementations, subsequent to computing the threenormalized Laplacian values (e.g., for all color channels, including R,G, and B), the computer system 100 may not compare one of the normalizedLaplacian values to a threshold, such as the blue color channel.

The computer system 100 can compare one or more of the normalizedLaplacian values to one or more thresholds respective of the colorchannels of the normalized Laplacian values. Based on the comparison,the computer system 100 can provide an alert to the user if the user'sfinger is not on the camera. For example, the computer system 100 candetermine or detect that a flesh (e.g., finger or other body parts) ispresent on the camera based on the following comparison:

$\{ {\begin{matrix}{\overset{\sim}{L}}^{red} & {< {\theta^{red}{and}}} \\{\overset{\sim}{L}}^{green} & {< \theta^{green}}\end{matrix}.} $

The computer system 100 can determine if the finger is not detected fora duration of consecutive frames. The duration of consecutive frameswith no finger detected can be denoted as T_(abs finger). The durationT_(abs finger) can be preset or preconfigured to a time duration, suchas 1 second, 0.5 seconds, 2 seconds, among others. If the finger (orother body parts) of the user is not detected for at leastT_(abs finger), the computer system 100 can flag the recording or thesequence of images of the finger as, e.g., isAbsentFinger condition(sometimes also referred to as finger abort condition, finger absentcondition, or ABSENT_FINGER condition). The computer system 100 canraise the finger absent condition if the computer system 100 does notdetect a finger for T_(abs finger) duration of consecutive frames. Insome implementations, the computer system 100 can raise the fingerabsent condition if the computer system 100 does not detect a finger fora total T_(abs finger) duration of non-consecutive frames.

For example, if T_(abs finger) duration is 1 second and the secondduration is 2 seconds, if the computer system 100 determines that atotal of 1 second out of the 2 seconds does not present the finger, thecomputer system 100 can raise the finger absent condition. Accordingly,the computer system 100 can output a boolean value (e.g., 1 or 0 basedon the comparison between the normalized Laplacian values and thethresholds and the T_(abs finger) duration) corresponding to the errorcondition (e.g., absent finger condition) to either trigger the absentfinger condition or proceed with determining other conditions ormeasuring the user's PPG. In some implementations, the absence of fingercondition may be redundant with the error state corresponding toabortion by finger removal. Thus, if the finger abort condition israised, the condition of absorption by finger removal may also beraised. In another example, if the finger abort condition is not raised,the condition of absorption by finger removal may also not be raised.

D.2. Finger Position Helper

One key factor for obtaining a good result for measuring blood pressure,heart rate/pulse rate, or other blood measurement is to have a goodplacement of a user finger, or other body parts for measuringcharacteristic of the blood stream, at the lens of the camera or ameasuring/capturing device. The lens of the camera can be in electricalcommunication with or a part of the computer system 100 or a mobiledevice 204. To assist users with the positioning of their finger toobtain good measurements, the computer system 100 executing theapplication 114 can process a video (e.g., one or more sequences ofimage frames) of the user finger captured by the camera.

An indication of whether the user finger is well positioned on the lensis to determine if all the pixels of images captured by the cameraevolve in the same way (e.g., similar color values or intensities (orconsistent changes in the color values or intensities) throughout theimage frames of a sequence of images). To determine such indication, thecomputer system 100 can measure the resemblance of pixels in the imagesby calculating the cross-correlation for one or more sequences ofimages.

The computer system 100 can measure the similarity between two or morevectors of the images by computing their cross-correlation. Thecross-correlation between a pixel and other pixels in an image (orbetween a vector and other vectors in a sequence of images) can bereferred to as a correlation matrix. The computer system 100 can storeintermediate sums between correlation matrices of one or more pixels,such that the intermediate sums between overlapping integrals are notcomputed multiple times, thereby reducing resource consumption andcomputation time. Further, the computer system 100 can aggregate thecross-correlation matrices (e.g., similarity matrix or distance matrix)to assess similarities (or differences) between each pixel and the restof the pixels in the downsampled image. The computer system 100 can usethe aggregate cross-correlation matrices to estimate a finger positionrelative to the center of the camera or photodetector lens. Accordingly,the computer system 100 can compute a gradient to assist users inorienting their fingers to cover the lens properly.

Referring to FIG. 18A, a flowchart illustrating a method 1800 ofdetermining a placement of a finger or body part is shown, according toexample embodiments. The method 1800 can include acquiring a sequence ofimages (STEP 1802). The method 1800 can include generating a sequence ofdownsampled images (STEP 1804). The method 1800 can include determining,for each pixel position of the downsampled images, a respectiveaggregate pixel similarity score (STEP 1806). The method 1800 caninclude determining, using aggregate pixel similarity scores of pixelpositions, a vector indicative of a position of a body part of a user(STEP 1808). The method 1800 can be performed by the computer system 100or the corresponding processor 102, for example, upon executing theapplication 114. The method 1800 can be performed by other components(e.g., memory 104, BUS 106, light source 108, camera device 110, ordisplay device 112) of the computer system 100, in conjunction with FIG.1 . The method 1800 can be performed by a remote device or a remoteserver. In some implementations, the method 1800 can be performed by acomputer system 100 and a remote server. For example, the computersystem 100 can establish a communication channel with the remote serverto forward data to or receive processed data from the remote server. Thecomputer system 100 can acquire the sequence of images and pass thesequence of images to a remote server. Responsive to receiving thesequence of images, the remote server can perform the steps 1804-1808.After determining the vector indicative of a position of a body part ofthe user, the remote server can transmit the processed data or thevector to the computer system 100 (e.g., for presentation to the uservia a display device 112). Accordingly, each step of method 1800 can beperformed by a combination of the computer system 100 and the remoteserver. The method 1800 can be performed by other components discussedherein and perform features and functionalities in conjunction withFIGS. 1-17 , for example. The method 1800 can include one or more stepsof or perform one or more similar steps to at least one of other methodsdiscussed herein, such as methods 400, 1200 or 1500, among other methodsdiscussed in this disclosure.

Still referring to FIG. 18A, in further detail, a photodetector of thecomputer system 100 can acquire a sequence of images (STEP 1802). Thephotodetector can be referred to as a photosensor, which can include orbe the light source 108. The computer system 100 can acquire thesequence of images by performing features or functionalities similar tostep 1602 discussed above in relation to FIG. 16 . For example, thesequence of images can represent transdermal optical data of a subject.A user can actuate the application 114 on the computer system 100, andapply their fingertip, finger pulp or other body parts to the camera 110for example, as illustrated in FIGS. 2A, 2B, and 3A. Upon initiation,the application 114 can cause the computer system 100 to display a userinterface (UI) on the display device 112. The UI can provideinstructions to the user to apply their fingertip against the camera 110or a corresponding transparent glass for a given period of time, e.g.,30 seconds. The UI can provide an option, e.g., upon user selection, topresent further instructions, images, or a demo explaining how thefingertip (or other body parts) is to be applied to the camera 110.

The application 114 or the data acquisition module 502 can automaticallytrigger the camera 110 to acquire a sequence of image frames. In someimplementations, the application 114 or the data acquisition module 502can cause the computer system 100 to display an interactive item as partof the UI. The application 114 or the data acquisition module 502 canautomatically trigger the camera device 110 to acquire the sequence ofimages, upon the user interacting with the interactive item. Triggeringthe camera device 110 can include actuating the flash device 206 to emitartificial light either continuously or periodically according to agiven frequency.

The camera device 110 can acquire the sequence of image frames while theuser's fingertip (or other body parts) is applied against the opticalwaveguide of the camera or the corresponding transparent glass.Specifically, the RGB sensor 312 can acquire the sequence of imageframes according to a specific or predefined frame rate based on thelight intensity I_(r) reflected from the user's finger. The frame ratecan be equal to 25 frames/second (fps), 30 fps, 50 fps or other framerate supported by the camera device 110. The data acquisition module 502can receive image data corresponding to image frames from the cameradevice 110. In some implementations, the data acquisition module 502 canreceive image data for each frame immediately after the image frame iscaptured or recorded by the RGB sensor 312 or the camera device 110.

In some implementations, the computer system 100 can include a pluralityof cameras 110. The computer system 100 can automatically select one ofthe cameras 110 to acquire the sequence of image frames, and display(e.g., via a user interface of the application 114) a notification tothe user of the computer system 100 of the selected camera 110. The usercan place his/her finger against the selected camera. The selection canbe based on the features or characteristics of the various cameras. Insome implementations, the user can select the camera to be used toacquire the sequence of image frames. The computer system 100 canreceive or obtain the user selection via an input device, e.g., akeyboard, a mouse, or a touch screen, among others, of the computersystem 100. The computer system 100 may confirm the user selection via,for example, the display device 112. The computer application 114 mayinclude different settings for distinct cameras 110. The computer system100 can maintain camera ID/phone model matrix in the memory 104. Besidesthe camera ID, the computer system 100 can store the preferred camerasettings in the matrix, such as ISO, exposure time, tone map, RGB gainor a combination thereof, among other settings

The computer system 100 can generate a sequence of downsampled images bydownsampling each image of the sequence of images (STEP 1804). Thecomputer system 100 can generate a sequence of downsampled images byperforming features or functionalities similar to step 1604 of FIG. 16 ,such as in conjunction with FIG. 9 . For example, the sequence of imagescan include an image region or block associated with at least one color(e.g., red (R), green (G), or blue (B)) frame 702. The data acquisitionmodule 502 of the application 114 executing on the computer system 100can decompose each image into a respective color (e.g., R, G, or B)frame. The computer system 100 can generate the sequence of downsampledimages responsive to or immediately after acquiring or obtaining theimages of the subject. In some cases, the computer system 100 candelegate the generation of the sequence of downsampled images to aremote server or a remote computing device. For example, the computersystem 100 can acquire and transmit the sequence of images to a remoteserver to process the sequence of images. Processing the sequence ofimages can include downsampling the sequence of images to generate asequence of downsampled images. The sequence of downsampled images canbe further analyzed or processed by the remote server or transmitted tothe computer system 100.

The computer system 100 can downsample each image of the sequence ofimages. The computer system 100 can perform the downsampling process inconjunction with FIG. 4 and FIG. 7 or execute the features orfunctionalities as described in FIG. 4 and FIG. 7 . For example, theprocessing module 504 of the application 114 executing on the computersystem 100 can identify an image region or block associated with a color(e.g. G or R) frame 702. The image can be from one of the sequence ofimages. The processing module 504 can downsample each color (e.g. G orR) frame 702 of the sequence of image frames to smaller image block 704.The processing module 504 can downsample the color (e.g. G or R) frame702 to a 5×5 image (or data) block, a 9×9 image (or data) block, or adata block 704 of a different size. In downsampling the color frame 702,the processing module 504 can apply low-pass filtering or averaging tothe pixel values of the color frame 702. The processing module 504 candownsample the color frame 702 using known downsampling techniques.

The processing module 504 can identify a sub-block 706 of the imageblock 704 for use to generate a color signal. The processing module 504can identify a central zone or sub-block of the image block 704 assub-block 706. Depending on the size and the placement of the userfinger against the camera, the color intensities of a central zone ofthe image block 704 (or the color frame 702) are more likely to reflectthe pulsatile nature of arterial blood in the finger than outer regionsof the image block 704 (or the color frame 702). In some cases, the userfinger may not be placed right at the center of the camera device 110,but rather shifted towards one direction over another. In such cases, atleast one outer region of the image frame may not adequately reflect thelight (or optical signal reflected from the user finger. In someimplementations, the identified sub-block 706 can be a 3×3 image (ordata) block, a 5×5 data block, or a data block of other sizes. Bydownsampling individual images of the sequence of images, computationalresources used to process the images, as well as resource consumptionfor storing the images that are not downsampled, can be reduced. Forexample, by downsampling images with thousands of pixels (e.g., 320×240,720×576, 1024×768, among other image sizes) to a 3×3 image, the computersystem 100 can significantly reduce computational resources used toprocess the downsampled images and resource consumption to store thedownsampled images.

The computer system 100 can determine, for each pixel position ofvarious pixel positions of the sequence of downsampled images, arespective aggregate pixel similarity score. The respective aggregatepixel similarity score can be indicative of a similarity of a pixel ofthe pixel position to pixels of other pixel positions over a time window(STEP 1806). The aggregate pixel similarity score can include or bereferred to as an aggregate penalty score associated with each pixelposition of the sequence of downsampled images or generally as asimilarity score aggregated for a respective pixel position. Thecomputer system 100 can compute the aggregate pixel similarity scoreindicating the similarity of a pixel of the pixel position to pixels ofother pixel positions over a time window. For example, the computersystem 100 can compare a pixel at a pixel position to other pixels foreach downsampled image. The computer system 100 can repeat the processfor all pixels of each downsampled image at different pixel locations.In some cases, the computer system 100 can compare a pixel at a pixellocation from a downsampled image with another pixel at the same pixellocation from another downsampled image at a different time, forexample.

To determine or compute the aggregate pixel similarity score Thecomputer system 100 can determine or compute a matrix containing thecross-correlation values for the reshaped input video. Subsequent todetermine the matrix, the computer system 100 can determine a singlescore (e.g., the aggregate pixel similarity score) for each pixel byaggregating multiple score values associated with the pixel position ofthe sequence of images. The computer system 100 can obtain or use avideo of the finger as input. The video can include or refer to asequence of images, denoted as V. The video V can have a shape or size(T, d_(x), d_(y), 3). The T can represent the sequence of images in V,which can be represented as the total duration or the total image framesof V. The d_(x) and d_(y) can represent the height and width of V or theimages of the sequence of images. For example, in downsampled images,d_(x) can equal to d_(y), which is equal to 5. The last variable in theshape of V can represent the total number of color channels, such as 3in this case. In some cases, the computer system 100 may use a singlechannel, such as green instead of red or blue, or red instead of greenor blue color channels, in which case V can have a shape or size (T,d_(x), d_(y)).

To reduce the dimension of V, the computer system 100 can reshape V to areshaped video, denoted as X (e.g., the reshaped version of V). Thereshaped video X can include shape (T, p) corresponding to vectorizationof each image from the 3-dimensional shape (d_(x), d_(y), 3) to a vectorof length p=d_(x)×d_(y)×3. In other words, p can represent the length ofa vector representing color intensity values of an image frame having a3-dimensional shape or size (d_(x), d_(y), 3) in the video (or sequenceof image frames) V. In some implementations, a single color channel(instead of all color cannels) can be used, in which case p=d_(x)×d_(y).In some cases, the reshaped video X can be zero padded to have the shape(T+n−1, p). With the zero padded shape of X, the computer system 100 cancalculate and output cross-correlation matrices having the same numberof timestamps as input video. The n can denote a window size used tomeasure the cross-correlation of each pixel position. The window sizecan be set to 1 second, 30 frames, among other duration and number ofimage frames. The window size can be configured, for example, by theoperator of the computer system 100, the application 114, or a remoteserver, if the images are being processed remotely. The window size caninclude or refer to a time window that the computer system 100 computesat least a sequence of pixel intensity values, color values, oraggregate pixel similarity score to determine the orientation andpositioning of the user finger. Further, p can denote the number ofpixels (e.g., size of each image) used for cross-correlation. If onlyone color channel is used for determining the finger position of theuser, p can be described as d_(x)×d_(y), such as only green, only red,or only blue.

The computer system 100 can determine or output a matrix includingcross-correlation values for respective pixel positions of the inputvideo X (e.g., the reshaped video). The matrix containing thecross-correlation values can be denoted as Γ. The matrix Γ can include ashape (T−n+1, p) or (T, p) if the input is zero-padded. To performcross-correlation between two vectors x and y, the computer system 100can use formula (11) for computation.

$\begin{matrix}{{{{cor}( {x,y} )} = \frac{{cov}( {x,y} )}{\sqrt{{V(x)} \cdot {V(y)}}}},} & (11)\end{matrix}$

where cov

$( {x,y} ) = {{\frac{1}{n}{\sum_{i = 0}^{n - 1}{x_{i} \cdot y_{i}}}} - {( {\frac{1}{n}{\sum\limits_{i = 0}^{n - 1}x_{i}}} ) \cdot ( {\frac{1}{n}{\sum\limits_{i = 0}^{n - 1}y_{i}}} )}}$

is the covariance of x and y, and V(x)=cov(x, x) is the variance of x.

The computer system 100 can cross-correlate each pixel position withother pixel positions in the sequence of images. Subsequently, thecomputer system 100 can construct the cross-correlation matrix of thereshaped video X by leveraging formula (12).

Γ[t;i,j]=cor(X[t−n+1:t,i],X[t−n+1:t,j]) with 0≤t<T, 0≤i, j<p.  (12)

As indicated above, the matrix containing the cross-correlation valuescan be denoted as Γ, which can be equated with the cor(X[t−n+1: t, i],X[t−n+1: t, j]). To further optimize the computation of thecross-correlation matrix, the computer system 100 can utilize thesymmetric nature of the cross-correlation matrix Γ as depicted inequation (13), which can be formulated (or recognized) from equation(12).

∀t,i,j,Γ[t;i,j]=F[t;j,i]  (13)

Considering the symmetric nature of the cross-correlation matrix Γdepicted in equation (13), the computer system 100 can compute half ofthe entries of the cross-correlation matrix and determine the secondhalf using the symmetric nature of the cross-correlation matrix Γ toobtain cross-correlation values for all pixel locations of the sequenceof images.

In some implementations, the computer system 100 can determine therespective sequence of pixel intensity values for each of various pixelpositions of the sequence of downsampled images. The respective sequenceof pixel intensity values can represent pixel intensities of the pixelposition over the time window. Each pixel position of the sequence ofdownsampled images can include their respective color values, colorintensities, among other features or characteristics representing thepixel. The computer system 100 can determine a cross-correlation valuerepresenting a cross-correlation between a pair of sequences of pixelintensity values representing pixel intensities of the pair of pixelpositions over the time window. The computer system 100 can determinethe cross-correlation value for each pair of pixel positions throughoutthe time window (or window size) of the video X. For example, thecomputer system 100 can cross-correlate pixel values (e.g., pixelintensity or pixel color) between a first pixel position and a secondpixel position, the first pixel position and a third pixel position, thefirst pixel position and a fourth pixel position, and so forth over thetime window. Based on the formula discussed above, the computer system100 may not re-compute or perform cross-correlation for pairs that havepreviously been cross-correlated to reduce computation resources. Withthe various cross-correlation values for each pixel position, thecomputer system 100 can aggregate the pixel similarity score using thecross-correlation values associated with each of the pixel positions. Toaggregate, the computer system 100 can perform the aggregate features,functionalities, and techniques discussed herein.

For optimization purposes, the computer system 100 can use the O(T×n×p²)operations to compute the complete cross-correlation matrix for thesequence of images for video X. The computer system 100 can use adynamic programming paradigm for optimizing the code or operationsdescribed above, which can consume more memory. Thus, by performing thisoptimization the complexity for computing the cross-correlation matrixfor each new frame can be reduced to O(p²).

In some implementations, the computer system 100 can store intermediatesums between the cross-correlation of vectors for each pixel of thesequence of images. For example, the cross-correlation between twovectors x and y can further be described in equation (14) for thereshaped video X as:

$\begin{matrix}\begin{matrix}{{\Gamma\lbrack {t,i,j} \rbrack} = {{cor}( {{X\lbrack {{t - n + {1:t}};i} \rbrack},{X\lbrack {{t - n + {1:t}};j} \rbrack}} )}} \\{{= \frac{C\lbrack {{t;i},j} \rbrack}{\sqrt{{C\lbrack {{t;i},i} \rbrack} \cdot {C\lbrack {{t;j},j} \rbrack}}}},}\end{matrix} & (14)\end{matrix}$

with C[t;i,j]=cov(X[t−n+1: t;i], X[t−n+1: t;j]) being the covariance ofpixels i and j on the time window ending at t. The covariance can berewritten or constructed as:

$\begin{matrix}\begin{matrix}{{C\lbrack {{t;i},j} \rbrack} = {{cov}( {{X\lbrack {{t - n + {1:t}};i} \rbrack},{X\lbrack {{t - n + {1:t}};j} \rbrack}} }} \\{= {{cov}( {X_{t}^{i},X_{t}^{j}} )}} \\{{= {\frac{1}{n}{\sum\limits_{t^{\prime} = {t - n + 1}}^{t}X_{t^{\prime}}^{i}}}},{X_{t^{\prime}}^{j} - {( {\frac{1}{n}{\sum\limits_{t^{\prime} = {t - n + 1}}^{t}X_{t^{\prime}}^{i}}} )( {\frac{1}{n}{\sum\limits_{t^{\prime} = {t - n + 1}}^{t}X_{t^{\prime}}^{j}}} )}}} \\{= {{\frac{1}{n}( {C_{t}^{i,j} - C_{t - n}^{i,j}} )} - {( {\frac{1}{n}( {S_{t}^{i} - S_{t - n}^{i}} )} ){( {\frac{1}{n}( {S_{t}^{j} - S_{t - n}^{j}} )} ).}}}}\end{matrix} & (15)\end{matrix}$

where C_(t) ^(i,j)Σ_(t′=0) ^(t)X_(t′) ^(i)·X_(t′) ^(j) is the cumulatedsum of the first cross products between pixels i and j, and where S_(t)^(i)=Σ_(t′=0) ^(t)X_(t′) ^(i) is the cumulated sum of the first pixel i.

By using these equations, the computer system 100 can reduce thecomplexity of the process by a factor of n. Further, to reduce resourceconsumption (e.g., memory space for storing the cumulated sums), thecomputer system 100 can drop or discard the cumulative sums for one ormore image frames that are outdated or beyond a predetermined duration,such as dropping cumulative sums stored before time instance t−n orassociated with time indices (e.g., subscript of S_(t-k) ^(i)) less thant−n. As such, the computer system can maintain only the last ncumulative sums S_(t-1) ^(i) to S_(t-n) ^(i) when moving from one imageframe to the next image frame.

The computer system 100 can compute a score to obtain a single value foreach pixel. The score can include or be referred to as an aggregatepixel similarity score, which aggregated the cross-correlation valuesdetermined for each of the pixel positions in the video X. Using theaggregated pixel similarity score, the computer system 100 can determineor compute a gradient (e.g., a vector) for presentation in a UI toassist users with their finger positioning. For example, the computersystem 100 can indicate how the user should orient or move their fingerto cover the lens, such as the center of the camera lens. The computersystem 100 can use at least one of V (e.g., video of the finger), X(e.g., reshaped version of V), and Γ (e.g., correlation matrix for eachpixel for at least one timestamp for simplicity) as inputs to computethe score. The correlation matrix Γ can include a shape with one of thethree color channels, such as only green. The shape of the correlationmatrix Γ can be (d_(x)×d_(y), d_(x)×d_(y)).

The computer system 100 can use an aggregate function to aggregatemultiple score values for one pixel in a single value. The aggregatefunction can be denoted as a in this case. The computer system 100 canuse a quantile, e.g., denoted q, for computing scores for individualpixels. An aggregation function can be the q-th quantile with q=0.8.With the inputs and parameters, the computer system 100 can determinethe orientation and position of the finger, as well as the distance fromthe optimal position (e.g., center) of the lens for blood measurement.Responsive to determining the orientation and position, the computersystem 100 can determine or generate a vector pointing or directing theuser to move the finger to where it should be, thereby properly coveringthe lens to obtain good measurements and results. Further, the computersystem 100 can provide a UI to guide users with their fingerpositioning, such as indicating where they move the finger, such thatpixels with lower scores (e.g., aggregate pixel similarity score) arebetter covered. The score can indicate the sufficiency of the reading ormeasurement of the blood pressure, pulse rate, among other healthfeatures of the user, at the pixel position associated with the score.In some cases, the score can represent a noise level (e.g., for adistance matrix—a high score can indicate a high noise level and a lowscore can indicate a low noise level), image quality, saturation level,among other indications of the quality of the sequence of images or thevideo. The configuration of whether a high score indicates poor qualityand a low score indicates high quality, and vice versa, is based on thepreference of the administrator of the application 114.

To determine which direction the user should move the finger, if notcovering the lens properly, the computer system 100 can determine orcompute an aggregate pixel similarity score for each pixel position ofthe sequence of images. As used herein, the aggregate similarity scorecan represent a measure of similarity or dissimilarity between a pixelposition and the rest of (or other) pixel positions. For example, thecomputer system 100 can use the aggregation function a for computing thepixel score. The computer system 100 can determine that if the score foreach pixel gets closer to zero, the pixel can be considered as goodquality. On the other hand, if the score is high, the pixel can beconsidered poor quality. In this case, the computer system 100 candetermine or compute a distance matrix, where a higher score canindicate a higher distance, higher variation, or having less correlationbetween pairs of pixel positions. In some implementations, the computersystem 100 can use a resemblance matrix, such as a cross-correlationmatrix, to determine the aggregate pixel similarity score for each pixelposition. The computer system 100 can compute the distance matrix as:

Δ_(j) ^(i) =Δ[i,j]=1−Γ[i,j]  (16)

The distance Δ[i,j] presented above may not have a specificdistribution. In other words, if two of the pixel positions greatlydiffer from each other, the computer system 100 may compute acorresponding high distance for the pixel positions. However, the actualvalue of the distance matrix may not be informative, for example, a highdistance between a pair of pixel positions and a very high distancebetween another pair of pixel positions may correspond to a similarsetup or result in the distance matrix.

The computer system 100 can use quantile estimation to determine thedistance of a pixel position to other pixel positions of the downsampledimage. For each pixel, the computer system 100 can compute the q-thquantile of the p distances of a pixel position to another. This yieldsto the aggregated penalty score (or aggregated similarity score) foreach pixel i, as:

π_(i)=α(Δ)[i]=Δ _((q·p)) ^(i)  (17)

Where Δ_((j)) ^(i) denotes the j-th element or entry of Δ^(i), where theentries of Δ^(i) are sorted in an ascending order. Therefore, by usingthe above formulas, the computer system 100 can determine a distancematrix and the aggregate pixel similarity scores for each of the pixellocations of the sequence of images.

In some implementations, the computing device can determine therespective aggregate pixel similarity score using the cross-correlationvalues associated with the pixel position. Responsive to determining therespective aggregate pixel similarity score, the computer system 100 candetermine, for each pair of pixel positions, a respective distance scoreusing the cross-correlation value. In this case, the respective distancebetween each pair of pixel locations can represent the cross-correlationbetween the pair of sequences of pixel intensity values. Further, thepair of sequences of pixel intensity values can represent pixelintensities of the pair of pixel positions over the time window. Forexample, the computer system 100 can cross-correlate between at leasttwo of the pixel positions having pixel intensity values associated withthe respective pixel positions. The computer system 100 cancross-correlate the pair of pixel positions over the time window. Thecomputer system 100 can repeat the cross-correlation for other pairs ofpixels of the sequence of images. By computing the cross-correlation,the computer system 100 can obtain or determine variouscross-correlation values associated with each respective pixel position.Therefore, for each pixel position, the computer system 100 candetermine the respective aggregate pixel similarity score as anaggregation of respective distance scores associated with the pixelposition. According to the above formula, the computer system 100 cancompute the negative of the aggregated cross-correlation values of eachpixel position to determine a distance matrix for each of thecorresponding pixel positions.

The computer system 100 can determine, using aggregate pixel similarityscores of various pixel positions of the sequence of downsampled images,a vector indicative of a position of a body part of a user (STEP 1708).The vector indicative of a position of a body part can include or referto an orientation vector. The body part can be a finger as referred toherein, for example. The computer system 100 can determine theorientation vector for assisting or guiding users to correctly placedtheir fingers by using formula (18).

$\begin{matrix}{\delta = \begin{pmatrix}\frac{\Sigma_{i}{x_{i} \cdot \pi_{i}}}{\Sigma_{i}\pi_{i}} \\\frac{\Sigma_{i}{y_{i} \cdot \pi_{i}}}{\Sigma_{i}\pi_{i}}\end{pmatrix}} & (18)\end{matrix}$

By using the orientation vector, the computer system 100 can derive adirection θ and an intensity I, such as in formula (19).

$\begin{matrix}{{I = {{❘\delta ❘} = \sqrt{\delta_{x}^{2} + \delta_{y}^{2}}}}{\theta = {{{angle}(\delta)} = {{atan}2{( {\delta_{y},\delta_{x}} ).}}}}} & (19)\end{matrix}$

Thus, by performing the techniques discussed above, the computer system100 can reduce the complexity of the process by a factor n, compute thecross-correlation between pairs of pixel positions (e.g., vectors), thedistance matrix, and the orientation vector to guide the user finger inan appropriate direction and distance for enhancing the quality of thebloodstream captured by the camera.

As depicted in equation (15), the computer system 100 can determine orcompute a center of mass of the aggregate pixel similarity scores of theplurality of pixel positions of the downsampled images, which may beviewed as representing an estimate of the position of the finger (orother body part) relative to a desired position. The smaller the vectorδ in magnitude, the closer is the position of the finger or body part tothe desired position, and the larger the vector δ in magnitude, thefarther is the position of the finger or body part from the desiredposition.

In some implementations, the computer system 100 can determine the angleof the vector δ indicative of a direction along which to move the fingeror body part of the user. For example, the computer system 100 candetermine that the finger of the user is not at the desired position onthe lens (e.g., the center of the lens). The computer system 100 candetermine the position of the finger based on the vector δ, where themagnitude of the vector δ can indicate how far (e.g., relatively) is thefinger from the desired position and the angle θ of the vector δ canrepresent the direction along which the finger position is deviatingfrom the desired position. Accordingly, the computer system 100 candetermine the trajectory or angle (e.g., the angle θ of the vector δ)along which the finger has to move in order to reach or overlap with thedesired position on the lens. In some implementations, the computersystem 100 can determine the magnitude of the vector δ indicative of arelative distance separating the finger (or body part) position and thedesired position. As such, the magnitude and angle of vector δ canprovide an indication for the user regarding in which direction and byrelatively how much to move the user's finger (or body part) to overlapthe desired position on the lens of the camera. The computer system 100can display an indication of the relative distance separating the fingerposition from the desired position on the lens, and/or an indication ofthe deviation angle θ. The farther the distance between the fingerposition and the desired position, the higher is the magnitude of thevector δ, as can be displayed on a UI for presentation to the user viathe display device 112. In another example, if the finger position isnear or in proximity to the desired position on the lens, the computersystem 100 will provide a low magnitude of the vector δ indicating asmall distance for the user to move their finger.

FIG. 18B shows a flowchart illustrating a method 1850 depicting anexample implementation of method 1800 of FIG. 18A. At step 1852, thecomputer system 100 can transform each acquired image frame into acorresponding one-dimensional vector as discussed above with regard tothe reshaping of the video sequence V. At steps 1854 and 1856, thecomputer system 100 can compute the cross-correlation matrix Γ asdiscussed above with regard to equations (13)-(15), and can compute thedistance matrix Δ as discussed above in relation to equation (16),respectively. At step 1858, the computer system can compute thesimilarity (or penalty) scores π, as discussed above with regard toequation (17). At steps 1860 and 1862, the computer system 100 canreshape the computed similarity (or penalty) scores π, into a matrixhaving a size (d_(x)×d_(y)), and compute the barycenter (or the centerof mass) vector δ of the matrix of the similarity (or penalty) scoresπ_(i) as discussed above with regard to equation (18), respectively. Atstep 1864, the computer system 100 can compute the magnitude |δ| of thebarycenter vector (or the relative distance between the finger positionand the desired position) and the angle θ of the barycenter vector δ asdiscussed above with regard to equations (19).

In some implementations, the computer system 100 can provide a visualoutput generated using the vector δ to guide the user to move theirfinger or body part relative to a center of the photodetector lens. Forexample, the computer system 100 executing the application 114 caninitiate a UI for display on the display device 112. The UI can includeelements indicative of at least the position of the user finger (or bodypart), the desired finger position on the lens, the quality of thefinger placement, and/or an indication or instruction regarding anyaction(s) to be taken by the user. The computer system 100 can displayvisual elements indicative of the desired position and the detectedposition of the user finger (or body part) on the lens of the camera. Ifthe detected finger position does not coincide or overlap with thedesired position on the lens (or the detected finger position is not“close enough” to the desired position), the computer system 100 candetermine or estimate a displacement between the detected fingerposition and the desired position, and display a visual representationor indication of the determined or estimated displacement. An exampleillustration of the visual outputs that may be displayed to the user isshown in FIG. 18C.

FIG. 18C shows examples of user interfaces (“UIs”) for assisting fingerpositioning of the user, according to example embodiments. The UIs canbe generated by the application 114 based on videos or a sequence ofimages captured by the camera 110 of the computer system 100. Thecomputer system 100 can present or display the UIs to the user via adisplay device 112. The UIs can include at least a first visual element1882 (e.g., a circle) indicative of a desired position for the fingerand a second visual element 1888 (e.g., a circle) indicative of thedetected finger position. The UIs can further include another visualelement 1886 (e.g., arrow or triangle) indicative of a direction formoving the user finger towards the desired position. In someimplementations, the direction for moving the user finger towards thedesired position can be inferred based on the relative positions of thefirst and second visual elements 1882 and 1884 (e.g., without the visualelement 1886). The desired position of the finger (or other body part)can be at the center of the lens of the camera or photodetector 110 (orother locations based on the configuration of the lens and/orconfigurations set by the administrator or operator of the application114 or the computer system 100).

The computer system 100 can display the UI (i) in FIG. 18C when the dataacquisition process is initiated but before the user finger is detected(or before sufficient image frames to detect the finger position areacquired). The computer system 100 can display the UI (ii) of FIG. 18Conce the finger position is detected as discussed above. The user canmove his (or her) finger towards the desired position, and the computersystem can display the UI (iii) to show an updated position of thefinger relative to the desired position. The UI (iv) of FIG. 18Cindicates that the detected finger position coincides or overlaps withthe desired position. While FIG. 18C, shows only four UIs, the computersystem 100 can display any number of UIs as the user finger moves. Insome implementations, the UIs can display an indication (e.g., text or acolor bar) of a finger placement quality.

The computer system 100 can determine or estimate the finger positionbased on the aggregate pixel similarity scores (as discussed above)and/or based on the cross-correlation values Γ[t,i,j]. The computersystem 100 can determine or detect the position of the user finger (orbody part) based on the highest aggregated similarity score for eachpixel position. Further, if the detected finger position does notcoincide or overlap with the desired position, the computer system 100can determine and output the magnitude and/or orientation of the vectorδ (or displacement) between the detected finger position and the desiredposition. The orientation can indicate which direction to move thefinger and the magnitude can indicate the distance to move the finger.The higher the magnitude, the more distance the finger should travel,and vice versa.

The computer system 100 can generate a finger placement quality meter(or color bar) indicating, for example, a quality of the user fingerplacement, a quality of user finger detectability and/or a quality ofthe acquired sequence of images. The computer system 100 can update thequality meter responsive to changes in the detected finger position,changes in the detectability of the finger and/or changes in the qualityof captured images. Once the gauge of the quality metric reached athreshold, the computer system 100 can indicate to the user, either atthe quality metric or other portion of the UI (e.g., via text), for theuser to maintain position or keep still. In other words, the computersystem 100 can notify the user that the measurement is in progress andfor the user not to move to maintain the stability of the measuredresults and avoid invalid or unstable measurement data. Accordingly, thecomputer system 100 can present the UI to assist users with their fingerpositioning.

The computer system 100 can notify the user of different states of thefinger, based on at least the detected positioning of the finger and/orthe pressure applied to the camera. The detected finger position iscorrelated to the quality of the signal or images that the computersystem 100 captures. The pressure applied on (or distance of the fingerfrom) the camera can affect the quality of the signal or images. Forexample, poor positioning and application of the finger on the cameramay result in the computer system 100 capturing a poor quality signal orimages. On the other hand, a well-positioned and application of thefinger on the camera 110 may result in the computer system 100 capturinga good quality signal or images. The computer system 100 candiscriminate between different finger placement states including, forexample, a well_placed_finger state, a misplaced finger state, and anunknown_position state of the finger. The computer system 100 candiscriminate and notify the user of the different states based on theaggregate penalty score (or aggregate discrimination score) for each ofthe pixel position in the sequence of images. The penalty score caninclude, refer to, or be based on the distance score and/or thecross-correlation values. For example, the computer system 100 candetect and/or output the well_placed_finger state or condition when mostor all (e.g., 80%, 85%, 90%, 95% or 100%, among otherratios/percentages) of the pixel positions have a low or relativelysmall aggregate penalty score (or aggregate similarity score), e.g.,compared to a first threshold score. The computer system 100 can detectand/or output the misplaced finger state when at least a portion (e.g.,35%, 30%, 25% of the total pixel positions, among otherratios/percentage) of the total pixel positions have a high or largeaggregate penalty score (or aggregate similarity score), e.g., comparedto a second threshold score. The computer system 100 can detect and/oroutput the unknown_position state when most or all (e.g., 70%, 75%, 80%,85%, 90%, 95% or 100%, among other ratios/percentages) of the pixelpositions of the image sequence have a high aggregate penalty score,e.g., compared to the second threshold score. The computer system 100can display a message indicating the state of the finger placement tothe user.

To determine the state, the computer system 100 can determine or obtainthe penalty score for each image frame captured by the camera 110. Theaggregate penalty score can refer to or include one or more scores(e.g., distance score or correlation score, in some cases) for each ofthe pixel positions in the image. The computer system 100 can determinean aggregate penalty score similar to determining the aggregate pixelsimilarity score or the distance score, such as discussed above inrelation with FIG. 18A. The penalty score matrix can be denoted as Δ,which can have a size of (d_(x), d_(y)).

The computer system 100 can use one or more parameters for determiningdifferent states of the finger. The parameters can include, for example,a penalty high quantile, a penalty low quantile, a penalty highthreshold, and a penalty low threshold. The penalty high quantile canrefer to a quantile used for estimating the highest penalty score for apixel. For example, the computer system 100 can use the high quantile ofthe penalty scores (or high quantile of aggregate similarity scores) toidentify images with detected finger placement or position close enoughto the desired position. The high quantile of the penalty scores can bedenoted as q_(high). The high quantile of the penalty scores (or theaggregate similarity scores) can be dependent on the penalty score used,and can be configured or preset to 100% or in some cases, lower than100%. The computer system 100 can use the low quantile of the aggregatesimilarity scores (or) for estimating the intermediate penalty score fora pixel. For example, the computer system 100 can use the low quantileof the penalty scores (or the aggregate similarity scores) to identifyimages where the detected finger position is relatively far from thedesired position to assist with the finger placement of the user. Thelow quantile of the penalty scores (or the aggregate similarity scores)can be denoted as q_(low). The low quantile of the penalty scores (orthe aggregate similarity scores) can also depend on the penalty scoreused, and can be configured or preset to 30% or other percentages (e.g.,either higher or lower).

The computer system 100 can use penalty thresholds to discriminatebetween pixel positions with “good” or relatively low penalty scores andpixels with “bad” or relatively high penalty scores. For example, thecomputer system 100 can use a high penalty threshold to identify pixelpositions considered to have “good” or low penalty scores. The highpenalty threshold can be denoted as β_(high), which can be dependent onthe penalty score used. The computer system 100 can be configured to use0.04 (e.g., among other values) for a cross-correlation-based penaltyscore. In another example, the computer system 100 can use a low penaltythreshold to identify pixel positions considered to have “bad” or highpenalty scores. The penalty low threshold can be denoted as β_(low),which can also be dependent on the penalty score used. The computersystem 100 can be configured with a low penalty threshold of 0.15 (e.g.,configurable among other values) for a cross-correlation-based penaltyscore. Based on any penalty score used, the q_(high) can be greater thanthe q_(low) and the β_(high) can be smaller than the plow.

As part of the steps for determining the state of the user finger orbody part (e.g., the position, pressure applied, or distance from thelens), the computer system 100 can compute or determine multipleaggregation metrics based on the penalty matrix Δ. The aggregationmetrics can include δ_(high)=quantile (Δ; q_(high)) and δ_(low)=quantile(Δ; q_(low)). By using the property that q_(high)>q_(low), the computersystem 100 can obtain δ_(high)>δ_(low). Responsive to computing theδ_(high) and q_(low), the computer system 100 can classify one or moreconditions of δ_(high) and δ_(low) with an associated state (sometimesreferred to as a finger helper state). The classification of theconditions can be as shown in Table 3 below.

TABLE 3 Classification of Finger States Condition Finger helper state$\{ \begin{matrix}\delta_{high} & {< \beta_{high}} \\\delta_{low} & {< \beta_{low}}\end{matrix} $ WELL_PLACED_FINGER $\{ \begin{matrix}\delta_{high} & {\geq \beta_{high}} \\\delta_{low} & {< \beta_{low}}\end{matrix} $ MISPLACED_FINGER $\{ \begin{matrix}\delta_{high} & {\geq \beta_{high}} \\\delta_{low} & {\geq \beta_{low}}\end{matrix} $ UNKNOWN_POSITION $\{ \begin{matrix}\delta_{high} & {< \beta_{high}} \\\delta_{low} & {\geq \beta_{low}}\end{matrix} $ Impossible case

The last case or condition in Table 3 is impossible. Assuming thatquantiles and thresholds meet the last condition of Table 3, then thequantiles and thresholds would satisfy:δ_(high)<β_(high)<β_(low)≤δ_(low)<δ_(high). As seen in this case, theresulting condition is impossible as it requires δ_(high)<δ_(high).

The computer system 100 can refer to the above classifications of thecondition to determine the state of the user finger placement, based onthe penalty scores (or aggregate similarity core) of the pixels of eachimage. The computer system 100 can output the classification as a visualillustration for the user. For example, based on the classification, thecomputer system 100 can provide at least one of the feedback to the user(e.g., via the application 114) indicating whether the finger is wellplaced (e.g., well_placed_finger state), misplaced (e.g., misplacedfinger state), or at an unknown location (e.g., unknown_position state).In the case of a misplaced finger, the computer system 100 can rely on avector with a direction and an amplitude, e.g., as discussed above withregard to vector δ. Using the vector, the computer system 100 canindicate which direction and how far the finger is from the center ofthe lens (or other desired/preconfigured position that the center massof the finger should be located). Therefore, the computer system 100 canhint the user to adjust their behavior for a better-placed finger on thecamera lens.

D.3. Pulse and Pulse Features Extraction

A photoplethysmogram (PPG) signal can be obtained by processing imagesof a finger (or other body parts) of the user. The images can becaptured using the camera 110, an RGB sensor (e.g., RGB sensor 312), orother optical devices. The computer system 100 can use the PPG signal tomeasure the blood pressure of the user based at least on changes in theblood volume. The computer system 100 can use the PPG signal to identifya succession of pulses. Further, the computer system 100 can split thePPG signal into individual pulses to analyze individual pulsesseparately. The computer system 100 can compute statistics based on thedistribution of the metrics over the individual pulses.

By splitting the PPG signal, the computer system 100 can analyze thefeatures and the quality of individual pulses (e.g., single pulseanalysis). The signals representing one or more pulses can be describedby their periodicity, such as a loop of a big ascending slope, followedby a slow descending slope. The computer system 100 can extractindividual pulses from the PPG signal by detecting successions of thepulse events or features in the high-pass-filtered PPG signal. Otherfilters can be used for the PPG signal to extract features and qualitiesof the pulses. The computer system 100 can obtain or capture one or morePPG signals that pulses can be extracted from. The PPG signal can bedenoted as a series (x_(k))_(1≤k≤N) of length N. The parameter k canrepresent the indices of the PPG signal values. The computer system 100can filter the PPG signal at a predetermined bandwidth. The bandwidthcan be denoted as [f_(low),f_(high)]. The bandwidth can be set to [0.4Hz, 3.5 Hz], among other frequency settings. The bandwidth can beadjusted to obtain an optimal setting for extracting at least one pulsefrom the PPG signal. Responsive to filtering the PPG signal, thecomputer system 100 can determine, identify, or extract features ofindividual pulses within the PPG signal to output a list of pulses. Thecomputer system 100 can determine the quality of the pulses based on thefeatures extracted from the PPG signal.

FIG. 19 shows a plot 1900 of an example PPG signal with signal features(also referred to as notable indices) used to identify separate pulsesof the PPG signal, according to example embodiments. The pulse events orpulse features used can include troughs (or peaks down) 1902, peaks 1904and zero-crossings up 1906 and zero-crossings down 1908.

Referring to FIG. 20A, a flowchart illustrating a method 2000 ofdetecting pulses in a PPG signal is shown, according to exampleembodiments. The method 2000 can include obtaining a PPG signalgenerated from a sequence of images acquired using a photodetector (STEP2002). The method 2000 can include detecting various signal features ofdifferent signal feature types of the PPG signal (STEP 2004). The method2000 can include determining, using the detected signal features, apredetermined (or predefined) sequence of signal feature types includedin a portion of the PPG signal (STEP 2006). The method 2000 can includeclassifying the portion of the PPG signal as a PPG pulse segment (STEP2008). The method 2000 can be performed by the computer system 100 orthe corresponding processor 102, for example, upon executing theapplication 114. The method 2000 can be performed by other components(e.g., memory 104, BUS 106, light source 108, camera device 110, ordisplay device 112) of the computer system 100, in conjunction with FIG.1 . The method 2000 can be performed by a remote device or a remoteserver. In some implementations, the method 2000 can be performed by acomputer system 100 and a remote server. For example, the computersystem 100 can be in communication with the remote server to delegateone or more tasks to the remote server. In this case, the remote servercan perform the one or more tasks, such as performing at least one ofSTEPS 2002-2008 and transmit an output to the computer system 100. Themethod 2000 can be performed by other components discussed herein andperform features and functionalities in conjunction with at least one ofFIGS. 1-19 , for example. The method 2000 can be performed inconjunction with, sequential, or prior to at least one of other methodsdiscussed herein, such as methods 400, 1200, 1600, etc.

Still referring to FIG. 20A, in further detail, the computer system 100can obtain a PPG signal generated from a sequence of images acquiredusing a photodetector (STEP 2002). The photodetector can include or beat least one of an RGB sensor 312 or a camera. For example, the computersystem 100 can capture a sequence of images (e.g., a video) of a fingerfor a period of N samples. The computer system 100 can obtain orgenerate the PPG signal using a sequence of images with a single colorchannel, such as only green or only red. In some cases, the computersystem 100 can obtain the PPG signal using a sequence of images withmultiple color channels (e.g., R, G, and/or B).

Prior to obtaining or generating the PPG signal, the computer system 100can process raw images captured using the photodetector. For example,the computer system 100 can generate a sequence of downsampled colorframes corresponding to the sequence of images by downsampling arespective color frame for each image of the sequence of images. Thecomputer system 100 can downsample the images to reduce the resolutionof the image for generating the PPG signal. By reducing the resolution(e.g., reduced to 5×5 pixels), the computer system 100 can decreaselatency and reduce resource consumption when processing the sequence ofdownsampled color frames. The downsampled color frames may include onlyone color channel, such as green or red. In some cases, the downsampledcolor frames may include multiple color channels.

Responsive to generating the sequence of downsampled color frames, thecomputer system 100 can identify, in each downsampled color frame of thesequence of downsampled color frames, a respective image blockrepresenting a central image region of the downsampled color frame andhaving a first size smaller than a second size of the downsampled colorframe. The respective image block can include or be referred to as aportion of the respective downsampled color frame. The central imageregion of the respective image block may be located at the center of thedownsampled color frame. For example, in a 5×5 pixels image (among othersizes), the computer system 100 can determine that the central imageregion is located at the inner 3×3 pixels of the downsampled colorframe. In another example, the computer system 100 can determine thatthe central image region is not located at the center or the innerportion of the downsampled color frame. Instead, the computer system 100can identify the center image region located edges of the downsampledcolor frame. The central image region may be of any size smaller thanthe downsampled color frame. For example, if the downsampled color frameis 10×10 pixels, the central image region can be 5×5 pixels, 5×9 pixels,7×4 pixels, among others.

The computer system 100 can generate a color intensity signal using therespective image blocks of the sequence of downsampled color frames. Forexample, the computer system 100 can identify, within the central imageregion of each respective image block, the color value or intensityvalue associated with image blocks. Based on the color value or theintensity value, the computer system 100 can generate a color intensitysignal having at least one of the color values or the intensity valueassociated with the color intensity signal. Accordingly, the computingdevice can generate the PPG signal using the color intensity signal todetermine a blood pressure value.

The computer system 100 can apply a filter on the PPG signal generatedusing the sequence of images. The filter can be a pass-band filter,among other filtering techniques, such as a low-pass filter, high-passfilter, etc. For example, the computer system 100 can input the PPGsignal into the pass-band filter set to predetermined low and highfrequencies thresholds (e.g., 0.4 Hz and 3.5 Hz as the lower thresholdand higher threshold, respectively). These thresholds can be referred toas cut-off frequencies [f_(low),f_(high)]. The computer system 100 canobtain or implement the pass-band filter by using two infinite impulseresponse (“IIR”) filters successively. The computer system 100 can applythe high-pass exponential filter causally and then anti-causally withcut-off frequency f_(low). Similarly, the computer system 100 can applythe low-pass exponential filter causally and then anti-causally withcut-off frequency f_(high). By applying the filter causally, thecomputer system 100 can generate an output that depends on past andcurrent inputs but not future inputs. By applying the filteranti-causally, the computer system 100 can generate an output thatdepends only on future input values. In other words, the computer system100 can apply the filters to PPG signals previously captured and PPGsignals that will be obtained by the photodetector. Accordingly, thecomputer system 100 can obtain a filtered PPG signal without highvariations. The global or general shape of the signal can be maintained,the signal can be centered around zero, and small irregularitiespresented in the signal can be filtered out.

The predetermined thresholds can be configured by the operator or theadministrator of the computer system 100, for example. By passing thePPG signal through the pass-band filter, the computer system 100 canfilter PPG signals below the low-frequency and above the high-frequencythresholds. Using the pass-band filter, the computer system 100 caneliminate quick variations of the signal to obtain a more stable andsmoother PPG signal.

The computer system 100 can detect various signal features of differentsignal feature types of the PPG signal (STEP 2004). The signal featurescan include or be referred to as notable indices. The computer system100 can extract the signal features from the PPG signal to identifyindividual pulses for extraction or division. To extract the features,the computer system 100 can compute one or more normalization featuresfrom the filtered PPG signal. Normalization features can refer tofeatures computed from a normalized signal, such as PPG signal centeredaround the zero. For example, the computer system 100 can compute aminimal prominence of the peaks (e.g., trough and high peak) of thesignal. The minimal prominence can be preconfigured to 5% the range ofvalues taken in input PPG x: p=0.05×(x_((95%))−x_((5%))). In this case,the range can be the difference between the 5^(th) and 95^(th)percentile of the signal values. The range can be configured to otherpercentiles of the signal values, such as between the 10^(th) and90^(th) percentile, 3^(rd) and 97^(th) percentile of the signal values,among other values. In some cases, the minimal prominence can beconfigured to the highest value and the lowest values of the signalvalues. The computer system 100 can compute the minimal pulse durationvia the following equation: d=0.5×f_(s), where f_(s) is the samplingfrequency.

Referring to FIG. 19 , the computer system 100 can extract some notableindices or feature types from the signal, such as the features 1902,1904, 1906 and 1908. The signal feature types can include at least apeak (sometimes referred to as peaks top), a trough (sometimes referredto as peaks down), a zero-crossing up, and a zero-crossing down. Thecomputer system 100 can detect other features obtainable from thesignal. The computer system 100 can extract peaks top in the signal witha prominence of at least p and a distance from peaks down of at least d.Similarly, the computer system 100 can extract peaks down in the signalwith a prominence of at least p and a distance from peaks top of atleast d.

The computer system 100 can identify or determine one or more points ofinterest, based on the extract features of different feature types. Theone or more points of interest can include at least the start of thepulse, a systolic peak, the start of the next pulse, the start ofreflective pulse, a diastolic peak, the end of a pulse, a systolic peakmax slope, a notch max slope, and a diastolic max slope. The start ofthe pulse can refer to an index of the start of a pulse or a time pointat which the PPG signal starts to increase (e.g., from the trough). Insome cases, the start of the pulse can begin at other locations withinthe slopes of the PPG signal, indicative of the pulse, such as at thesystolic peak. The systolic peak can refer to an index at which thesignal reaches its maximum value. The systolic peak can include or bethe peaks up of the signal. The start of the next pulse can be an indexat the next time point similar to the start of the pulse. The start ofthe next pulse can be referred to as a start of a second pulse. Thestart of a reflective pulse can be an index after the systolic peak atwhich signal second derivative reaches its maximum. The diastolic peakcan be an index at which the signal second derivative reaches itsminimal value after the reflective pulse. The end of a pulse can be anindex at which the slope of the signal has come back to a stationarystate. Depending on the start of the pulse (e.g., the index positionstarting the pulse), the end of the pulse can be located before thestart of the next pulse, but after the diastolic max slope. The systolicpeak max slope can be an index of maximal slope in the PPG signalbetween the start of the pulse and the systolic peak. The notch maxslope can be an index of maximal slope in the PPG signal after thedicrotic notch. The diastolic max slope can be an index of the maximalslope of the diastolic descent within the PPG signal. With at least theaforementioned pulse features, the computer system 100 can identify oneor more pulses from the PPG signal generated using a sequence of images.

The computer system 100 can determine, using the detected signalfeatures, a predetermined (or predefined) sequence of signal featuretypes included in a portion of the PPG signal (STEP 2006). Thepredetermined sequence of signal feature types can be used to indicate,describe, or in some cases, define PPG pulse segments. The sequence ofsignal feature types can be predetermined by the operator oradministrator of the computer system 100. The predetermined sequence ofsignal feature types can include a ordered sequence of feature types,such as a first peak (e.g., peaks up) 1904, a zero-crossing down 1908, atrough (or a peak down) 1902, a zero-crossing up 1906 and a second peak(e.g., a second peaks up) 1904. In some cases, the predeterminedsequence of signal feature types can include an ordered sequence of afirst trough 1902, a zero-crossing up 1906, a peak 1904, a zero-crossingdown 1908, and a second trough 1902. The predetermined sequence ofsignal feature types can start at any of the feature types and end atthe next feature of the same feature type.

In some implementations, the ordered sequence of feature types may notinclude a second peak. For example, the computer system 100 candetermine a predetermined sequence of signal feature types of thetrough, zero-crossing up, peak, zero-crossing down, and an indexindicating the end of the pulse. In some implementations, the computersystem 100 can determine a predetermined sequence of signal featuretypes in other portions of the PPG signals, such as after the secondpeak or second trough.

In some implementations, the portion of the PPG signal indicative of apotential pulse can be a first portion that ends at or before a firstsignal feature (e.g., index of the start of the next pulse). Thecomputer system 100 can determine, using the detected signal features,the predefined sequence of signal feature types in a second portion ofthe PPG signal. The second portion can be different from the firstportion of the PPG signal and may include one or more feature typesdifferent from the first portion. The second portion of the PPG signalcan start at the first signal feature. The computer system 100 canextract one or more features from the second portion for classification.Based on whether the features of the second portion follow or include anordered sequence of feature types similar to the predetermined orderedsequence, the computer system 100 may classify the second portion of thePPG signal as a second PPG pulse segment.

The computer system 100 can classify the portion of the PPG signal as aPPG pulse segment based on the predetermined ordered sequence of signalfeature types included in a portion of the PPG signal (STEP 2008). Forexample, after extracting the features of different feature types fromthe signal, the computer system 100 can determine the sequence of thefeature types within the portion of the PPG signal. If the sequence offeature types identified in the signal matches the predetermined orderedsequence of signal feature types, the computer system 100 can classifythe portion of the PPG signal as a pulse. In some cases, if a firstportion of the PPG signal does not satisfy the predetermined orderedsequence, the computer system 100 can extend the first portion of thePPG signal until the remaining feature types are included in the firstportion. In other words, the computer system 100 can keep identifyingfeatures successively and extending the first portion to include theinclude additional identified features until the first portion includesor satisfies the predetermined ordered sequence of signal feature types.The computer system 100 can then classify the first portion of the PPGsignal as a pulse.

Referring to FIG. 20B, various examples of identified pairs of pulsesare shown, according to example embodiments. The computer system 100 canextract pulses in successive pairs (e.g., multiple portions of thesignal), such as to bypass any phase shift in the pulses. The successivepairs can refer to multiple portions of the signal, which may beclassified as two pulses, a single pulse, or no pulses (e.g., in caseswhere the ordered sequence is not satisfied). Using successive pairs,the computer system 100 can extract the pulse starting at any featuretype, such as the peak 1904 instead of the trough 1902. With successivepairs, the computer system 100 can extract the whole pulse starting atany feature type, instead of just part of the pulse, for example, whenthe extraction starts from the peaks 1904. The computer system 100 (or acorresponding computing device) can screen out feature types to detectthe successions of at least i) peak down or trough 1902, ii)zero-crossing up 1906 (e.g., a point where negative value turnspositive), iii) peak up or peak 1904 (e.g., the peak or summit of theportion of the signal), iv) zero-crossing down 1908 (e.g., a point wherepositive value turns negative), v) trough 1902, vi) zero-crossing up1906, vii) peak 1904, viii) zero-crossing down 1908, and ix) trough (orpeak down) 1902 again. In this case, the computer system 100 canidentify the successive pairs of pulses starting at the trough (or peakdown) 1902. The computer system 100 can identify each pulse as thepredefined sequence of feature types: trough 1902, zero-crossing up1906, peak 1904, zero-crossing down 1908 and trough 1902 again. Thecomputer system 100 can identify the successive pairs of pulses startingat any of the feature types discussed herein. For example, the pairs ofpulses (or pulses) can start and end with peaks 1904, the pairs ofpulses can start and end with zero-crossings up 1906, or the pairs ofpulses can start and end with zero-crossings down 1908. The computersystem 100 can use similar logic to identify other pulses (or otherpairs of pulses) within the PPG signal by successively identifyingfeatures of the different feature types until the predefined sequence offeature types is identified for each new pair of pulses.

In some implementations, in identifying the predetermined orderedsequence of feature types, the computer system 100 can identifyadditional features in between due to irregularities in an identifiedpulse or an identified pair of pulses. The dashed circles in FIG. 20Brefer to such irregularities or additional features. For example, thepair of pulses in the top left corner of FIG. 20B include the featuresof a zero-crossing up, a peak and a zero-crossing down (shown within thedashed circle) as additional or extra features that are identifiedbetween a zero-crossing down and a trough of the predetermined orderedsequence of feature types. In other words, the pair of pulses includes12 features instead of nine features. Referring to the second and thirdplots in the second row of FIG. 20B, each of these pairs of pulsesincludes an extra two features, namely an extra zero-crossing down andan extra zero-crossing up. Both pairs of pulses include 11 featuresinstead of nine features. Other irregularities are depicted in the thirdand fourth plots in the third row of FIG. 20B. When facing or detectingsuch irregularities, the computer system 100 can continue searching forthe next features in the PPG signal until the predefined orpredetermined ordered sequence of feature types is identified. In otherwords, the computer system can ignore the extra features (or extrafeature types) and keep looking for the missing features in thepredefined or predetermined ordered sequence of feature types.Accordingly, some of the identified pulses or pairs of pulses mayinclude more features than those corresponding to the predefined orderedsequence of feature types.

The computer system 100 can determine the pulse duration by computing adifference between two timestamps corresponding to the respective twofeature types (e.g., same family of feature type), such as two of thesame feature types. In some cases, the computer system 100 can calculatethe duration of the pulse as a difference between a first timestamp of afirst index representing the start of the pulse and a second timestampof a second index representing the end of the pulse. The second indexcan also represent the start of the next pulse.

In some implementations, the computer system 100 can compute, using thevarious signal features of at least one of the different signal featuretypes of the PPG signal, the characteristics of the pulses of the user.For example, the computer system 100 can determine the followingcharacteristics of the pulses: duration between time points (e.g.,indicating a pulse or a heartbeat), differences of amplitudes, relativedifferences of amplitudes, slopes and relative slopes at one or morefeature types, slopes and relative slopes between different featuretypes, and accelerations and relative accelerations at one or morefeature types. The computer system 100 can compute other characteristicsusing the extracted features of the PPG signal. These characteristicscan be used to determine the quality of the pulse or the health of theuser. In some implementations, the pulses discussed above can bereferred to as different PPG pulse segments corresponding to therespective portions of the PPG signal. The computer system 100 canclassify other portions of the PPG signal as different PPG pulsesegments based on the ordered sequence of feature types.

In some implementations, if the portion (e.g., PPG pulse segment or anindication of a pulse) of the PPG signal starts at a first peak 1904 andends at a second peak 1904 (e.g., the peaks up), the computer system 100can determine a duration of the PPG pulse segment based on a length oftime between the first peak and the second peak. If the portion of thePPG signal starts at a first trough and ends at a second trough, thecomputer system 100 can determine the duration of the PPG pulse segmentbased on the length of time between the first trough and the secondtrough. The duration of the PPG pulse segment can reflect the durationof a pulse. In some cases, the duration of the PPG pulse segment may notindicate just the duration of a pulse but also the duration of the pulseand the stable state of the signal (e.g., after the heartbeat) up untilthe beginning of a second pulse.

In some implementations, the computer system 100 can generate, from thePPG signal, one or more PPG pulse segments indicative of at least therespective one or more pulses. Each PPG pulse segment can include signalfeatures (e.g., at least one peak, at least one trough, at least onezero-cross up, and at least one zero-crossing down). The signal featuresin the PPG pulse segment can include the predetermined ordered sequenceof signal feature types. The predetermined ordered sequence of featuretypes can be configured by the administrator of the computer system 100.

In some implementations, the computer system 100 can determine that atime duration between a first feature of a first type and a secondfeature of a second type within a PPG pulse segment exceeds a timeduration threshold. For example, the first type can be a trough and thesecond type can be a peak up. The computer system 100 can identify atime duration threshold associated with the first type and the secondtype of features. The time duration threshold can indicate either aminimum time or a maximum time between the first type and the secondtype for a normal pulse. If the duration between the first type and thesecond type does not satisfy at least one of the minimum time or themaximum time, the computer system 100 can set a flag (e.g., indicating acorrupt or bad pulse) to the PPG pulse segment, such as responsive todetermining that the time duration exceeds the time duration threshold.

A physiological pulse has several characteristics that distinguishes itfrom noise or false positives. For instance, the slope at azero-crossing downward is very steep, implying that the duration betweena peak up and a zero-crossing down is very short. Accordingly, thecomputer system 100 can check whether the following inequalities aresatisfied:

$\{ {\begin{matrix}{{30{ms}} \leq {{{zero} - {crossing}{down}_{0}} - {peak}_{0}} \leq {250{ms}}} \\{{30{ms}} \leq {{{zero} - {crossing}{down}_{1}} - {peak}_{1}} \leq {250{ms}}}\end{matrix},} $

where the zero-crossing down₀ and zero-crossing down₁ representsuccessive zero-crossings down of the notable indices (or features ofthe predefined ordered sequence of feature types), and the peak₀ andpeak₁ represent successive peaks (or peaks up) of the notable indices(or features of the predefined ordered sequence of feature types). Ifany of the above inequalities is not satisfied, the computer system 100can declare the corresponding pulse segment as a corrupted pulse or canreject the pulse segment as not a valid pulse. In some implementations,the computer system 100 can use other inequalities and/or durationsbetween other features to determine whether a pulse segment is corruptor a valid pulse.

Referring to FIG. 20C, examples of a corrupted pulse and valid or good apulse are shown, according to example embodiment. The plot on the leftside illustrates an example of a corrupted pulse, while the plot on theright side of FIG. 20C illustrates an example valid or good pulse.

In some implementations, the first type and the second type can be othertypes of features, such as zero-crossing up or zero-crossing down. Inthis case, the computer system 100 can compare the time between thesefeatures to a second time duration threshold, different from theaforementioned time duration threshold for a trough and a peak featuretypes. In some implementations, if the time duration between the firstfeature and the second feature satisfies the time duration threshold,the computer system 100 may not set a flag to the PPG pulse segment(e.g., as corrupted or not valid pulse).

In some implementations, the computer system 100 can determine theduration of the PPG pulse segment based on the length of time between afirst signal feature and a last signal feature of the predetermined (orpredefined) sequence. The first signal feature and the last signalfeature of the predetermined sequence can be the same feature type. Insome other cases, the first signal feature and the last signal featuremay not be the same type, but rather, the last signal feature can be afeature type before the start of the next pulse. Similar to the above,the computer system 100 can compare the duration between the firstsignal feature and the last signal feature of the predetermined sequenceto a time duration threshold. Based on the duration of the PPG pulsesegment not satisfying the time duration threshold, the computer system100 can set a flag to the PPG pulse segment. Otherwise, the computersystem 100 can proceed to analyze the next PPG pulse segment forirregularities or to measure the blood volume or pulses of the user.Accordingly, by performing the aforementioned techniques, the computersystem 100 can compute raw estimates of the duration of individualpulses, the location of where the pulses are measured from (e.g., partof the user body), and the quality of the individual pulses. The qualityof individual pulses can refer to the consistency of the pulses, thelength of each pulse, the pulse rate, among other characteristics of thepulse, which can be used, in part, as an indicator of the user's health.By extracting different features from the PPG signal, the computersystem 100 can improve the precision of measuring or computing thecharacteristics and qualities of individual pulses.

Individual pulses can carry information about the blood pressure orother vital conditions of the user. By using the pulse division formulaor algorithm, the computer system 100 can identify or divide the signal(e.g., PPG signal) into multiple pulses (sometimes referred to as roughpulses). In some implementations, the signal may be corrupted, may notbe of good quality, or may not contain precise features, e.g., caused byan algorithm processing the signal. In this case, the computer system100 can use or rely on one or more pulse positions (e.g., from theoriginal signal as a start) for refining the pulse features and removingthe corruption.

The computer system 100 can refine the pulse features herein, using awindow (e.g., small window) in the PPG signal by using pulse divisionindices, such as discussed in at least one of FIG. 20A or FIG. 21 . Byusing the small window in the PPG signal, the computer system 100 canidentify notable indices (sometimes referred to as signal features) withbetter accuracy using raw PPG signal. The computer system 100 canperform one or more steps herein to refine the pulse or signal features.The one or more steps can be performed either sequentially,consecutively, in parallel, or in any other order, for example.

The computer system 100 can prepare the pulse for boundary extraction(e.g., indications of where individual pulses start or stop). Forexample, as a first step, the computer system 100 can gather a number offrames used for cropping the PPG signal around the start of a pulse.This number of frames can be denoted as n. The computer system 100 mayset n as 2×f_(s) (e.g., two times the sampling frequency f_(s)) framesof the original PPG signal, among other values. The start of thegathered n frames of the original PPG signal can be set as an offset ofa window for cropping the PPG signal. In this case, the offset can bereferred to as l herein. The start of the gathered n frames can start atl frames before the rough pulse start. The rough pulse start or therough start of the pulse can be or include the general area where thepulse starts or divides from other pulses. For example, the rough pulsestart can be located at the trough, the systolic slope, among others.The l can be configured or preset at 0.3 seconds (e.g., 0.3×f_(s)),among other offset times of the window, before the rough start of thepulse.

Subsequent to gathering n frames, the computer system 100 can upsamplethe PPG signal to a target frequency. The target frequency can be thefrequency at which the PPG signal should be upsampled, which can bedenoted as f_(up). The f_(up) can be configured or preset to 500 Hzamong other frequencies for upsampling the PPG signal, such as greaterthan or less than 500 Hz. The target frequency can be configured with aninterpolation of order 6. In some cases, the target frequency can beconfigured with other interpolation orders. By upsampling the signal,the computer system 100 can increase the smoothness of the signal andthe first derivative of the signal. Responsive to upsampling the signal,the computer system 100 can compute the first two derivatives of thesignal. Accordingly, the computer system 100 can prepare the pulse forthe extraction of pulse boundaries.

As shown in FIG. 22B, a plot of an example PPG signal can beillustrated, according to example embodiments. The plot can includevarious original pulses (e.g., data points of the PPG signals) forgraphing the various pulses. With the various original pulses or datapoints, the computer system 100 can generate the upsampled pulse,presented as the continuous line in this example. The example plot canthe PPG signal upsampled to 500 Hz with a b-spline interpolation in theorder of 6. The computer system 100 can determine, identify, orconfigure the rough pulse start at the systolic slope positioned at 10frames (or 333 ms for a frame rate of 30 fps). The computer system 100can use or refer to plot for extracting the pulse boundaries.

After upsampling and computing the successive derivatives (e.g., thefirst and the second derivative of the PPG signal), the computer system100 can find the indices of pulse boundaries. FIG. 22C illustrates anexample extraction of the pulse boundaries (or the indices of the pulseboundaries) according to example embodiments. The upsampled PPG signalcan be illustrated in a first subplot. Further, a second subplot and athird subplot can represent examples of the first and second derivativesof the upsampled PPG signal of the first subplot, respectively. To findthe indices of the pulse boundaries, the computer system 100 can beconfigured with one or more criteria for identifying one or more indicesof the pulse boundaries. For example, the start of the pulse (e.g.,which can be the first trough of the PPG signal) can include a height of90% of the signal amplitude (e.g., lower height or minimum height). Thesignal amplitude can refer to the averaged, among other aggregationtechniques for the amplitudes of the signal throughout the signal, forexample. The systolic peak of the pulse (e.g., the first peak of thesignal) can include a height of 90% of the signal amplitude (e.g., upperheight or maximum height). The start of the next pulse can be an indexlocated at the second trough with a similar height as the first trough(e.g., 90% of the signal amplitude). The computer system 100 can extractother pulse boundaries as discussed herein, based on the configurationor the definition of individual indices (e.g., the percentage or ratioof the signal amplitude, among other points of the signal). Theextracted pulse boundaries can be extracted and marked according to theexample illustrations of FIG. 22C.

The computer system 100 can extract one or more characteristic featuresof the pulse responsive to extracting the pulse boundaries or aftersegregating individual pulses. For example, the computer system 100 canextract at least the notch (sometimes referred to as a pulse notch), thediastolic peak (sometimes referred to as a pulse diastolic peak), andthe end of the pulse (sometimes referred to as pulse end) of the PPGsignal. For example, the notch can be described as the index between thesystolic peak and the start of the next pulse where the secondderivative reaches its maximal value. Extraction of the pulse notch canbe illustrated in the example plot in conjunction with FIG. 22D. Thediastolic peak can be described as the index between the notch and thestart of the next pulse where the second derivative reaches its minimalvalue. Extraction of the pulse diastolic peak can be illustrated in theexample plot in conjunction with FIG. 22E. The end of the pulse can bedescribed as the index between the diastolic peak and the start of thenext pulse where the second derivative reaches its maximal value.Extraction of the pulse end can be illustrated in the example plot inconjunction with FIG. 22F. The computer system 100 can extract othercharacteristic features of the PPG signals discussed herein.

Responsive to extracting the characteristic features, the computersystem 100 can determine the characteristic slopes based on thecharacteristic features. FIG. 22G illustrates examples of thecharacteristic slopes in the 0^(th), 1^(st), and 2^(nd) derivatives ofthe upsampled PPG signal. To determine the slope, the following indicescan be described for the computer system 100 to calculate the slope. Forexample, the systolic max slope index can be described as the indexbetween the start of the pulse and the systolic peak where the firstderivative reaches its maximal value. The notch max slope index can bedescribed as the index between notch and diastolic peak where the firstderivative reaches its maximal value. The diastolic max slope index canbe described as the index between the diastolic peak and the end of thepulse where the first derivative reaches its minimal value. Based on thedescription of the aforementioned slope indices, the computer system 100can determine the characteristic slopes based on the characteristicfeatures. The characteristic slopes can be illustrated, flagged, ormarked as shown in the one or more subplots of FIG. 22G.

The computer system 100 can augment the features for refining thefeatures of the PPG signal or pulses within the PPG signal. For example,the computer system 100 can obtain or determine the cross features orrelative features by computing different combinations between thecharacteristic slopes or characteristic features, such as F_(j)=f_(j)(t₁, x₁, t₂, x₂, . . . , t_(n), x_(n)), where F_(j) can denote a crossfeature, f_(j) can denote a function for aggregating different primaryfeatures and indices into cross features, and t_(i) and x_(i) can denoteindices and corresponding feature values (e.g., PPG, slope, etc.),respectively. For example, to compute the systolic peak duration, thecomputer system 100 can use the following formula: Peakduration=t_(sys peak)−t_(pulse start). The computer system 100 may useone or more features (e.g., primary features) in the cross feature. Insome implementations, the computer system 100 may not use certainfeatures in the cross feature. Accordingly, by performing the refinementprocess, the computer system 100 can efficiently obtain an improvedestimate of the blood pressure, heart rate, among other vital signs ofthe user based on the features or characteristics of the PPG signal.

D.4. Pulse Based Conditions

To measure a PPG or a PPG signal, a finger (or other body parts) shouldbe present on the camera lens or other photodetectors for capturingimages of the finger. However, in certain cases, the user finger may notbe present on the lens, the user may not be still (e.g., moving around),the user may remove their finger before the computer system 100 finishwith analyzing the blood volume of the user based on the PPG signals, orother conditions may arise that would degrade the quality of thecaptured images or data for processing by the computer system 100.Therefore, to make sure that the recording of the finger is performedunder expected conditions, the computer system 100 can analyze a colorintensity signal (or PPG signal) generated from the sequence of imagesto determine if, for example, the finger is on the lens or the user iskeeping still while performing blood volume measurement.

The computer system 100 can obtain, receive, or identify one or morepulses within the PPG signal. The computer system 100 can identify oneor more pulses from a list of pulses. Each pulse can be associated witha portion or a PPG pulse segment of the PPG signal. The computer system100 can determine the duration of individual pulses within a sequence ofpulses as discussed in at least STEPS 2006 and 2008 in conjunction withFIG. 20A, for example. The list of pulse durations can be denoted as(D_(i))_(i≥0) herein. The index i in this case can refer to or representthe pulse index of a pulse whose pulse duration is denoted D_(i).

The computer system 100 can use one or more parameters to determine oneor more conditions that can be used to alert the user. In some cases,the computer system 100 can use the parameters to determine if at leastone error condition is present. The one or more parameters can includeat least a consecutive duration ratio outlier quantile, a consecutiveduration ratio threshold, an overall duration quantile, and a durationconsecutive ratio threshold. A quantile can refer to or indicate a cutpoint dividing between value ranges. For example, a 5% quantile candivide or separate the maximum or minimum 5% of all values from the restof the values. The consecutive duration ratio outlier quantile can referto a quantile used for eliminating extreme duration ratios between twoconsecutive pulses, denoted as q_(cdr). The duration between twoconsecutive pulses can refer to a time duration between the start of apulse and the end of the pulse for two consecutive pulses. In somecases, the time duration can be from the start of a first pulse to thestart of the second pulse. Using q_(cdr), the computer system 100 candetermine the magnitude of variation in pulse duration for consecutivepulses or consecutive pairs of pulses. The ratio between two consecutivepulses can be calculated by dividing a first pulse duration of a firstpulse by a second pulse duration of a second pulse where the first andsecond pulses are consecutive pulses. The computer system 100 caneliminate, discard, or ignore ratios above a maximum quantile or below aminimum quantile, for example.

In some implementations, the q_(cdr) can indicate whether the user ispositioning the finger (or body part) properly on the lens. The q_(cdr)can provide indication of other conditions that, for example, contributeor lead to the extreme (or relatively large) values of q_(cdr). In somecases, the computer system 100 can notify or ask the user to readjustthe position of their finger or remain still during the measuringprocess, based on determined values of the q_(cdr).

The consecutive duration ratio threshold can refer to or represent athreshold value for pulse duration ratios for pairs of consecutivepulses, which can be denoted as θ_(cdr). If at least one computedduration ratio for a pair of consecutive pulses is above θ_(cdr), thecomputer system 100 can raise the ARRHYTHMIA condition, indicating thatthe user may potentially have arrhythmia based on the characteristics ofthe pulses (e.g., the durations of subsequent pulses).

The overall duration quantile can refer to a quantile used foreliminating or identifying extreme pulse durations within a recorded PPGsignal, denoted herein as q_(od). The computer system 100 can measure orcompute the pulse durations as discussed above in subsection D.3. Insome implementations, the computer system 100 can compute or determineextremal pulse durations using the overall duration quantile, asdiscussed in further detail below. The computer system 100 can use theextremal pulse durations to detect additional conditions. For instance,the computer system 100 can use the extremal pulse durations to detectpresence of an irregular heart rate condition. Specifically, a relativelarge difference or relatively large ratio between the extremal pulsedurations can be indicative of the irregular heart rate condition. Thecomputer system 100 can compare both extremal pulse durations of a PPGsignal to detect irregular heart rate.

The computer system 100 can compute or determine a maximal durationratio representing a ratio between the extremal pulse durations of thePPG signal. This computer system 100 can use a maximal duration ratiothreshold denoted as θ_(odr). The maximal duration ratio thresholdθ_(odr) can be configured or predefined by the administrator of thecomputer system 100. If the computer system 100 determines that themaximal duration ratio is greater than the threshold θ_(odr), thecomputer system 100 can raise an IRREGULAR HEART RATE conditionindicating that the user has irregular heart rate. The conditions raisedcan be displayed on a display device 112 of the computer system 100 aspart of the UI of the application 114, for example. Based on at leastone of the thresholds above, the computer system 100 can output aboolean value corresponding to one or more error conditions, such asARRHYTHMIA or IRREGULAR HEART RATE. These error conditions can betransmitted to the administrator or used by the computer system 100 totrain a machine learning model for enhancing the accuracy of identifyingerror conditions discussed herein.

Referring to FIG. 21 , a flowchart illustrating a method 2100 ofdetermining one or more error conditions based on features of pulsesegments of the PPG signal is shown, according to example embodiments.The method 2100 can include identifying, within a PPG signal generatedfrom a sequence of images acquired using a photodetector, various PPGpulse segments, each of the PPG pulse segments identified using apredetermined (or predefined) sequence of signal feature types (STEP2002). The method 2100 can include determining, for each PPG pulsesegment, one or more respective pulse features (STEP 2004). The method2100 can include determining one or more conditions based on the one ormore respective pulse features of the various PPG pulse segments (STEP2006). The method 2100 can be performed by the computer system 100 orthe corresponding processor 102, for example, upon executing theapplication 114. The method 2100 can be performed by other components(e.g., memory 104, BUS 106, light source 108, camera device 110, ordisplay device 112) of the computer system 100, in conjunction with FIG.1 . The method 2100 can be performed by a remote device or a remoteserver. In some implementations, the method 2100 can be performed by acomputer system 100 and a remote server. For example, the computersystem 100 can be in communication with the remote server to delegateone or more tasks to the remote server. In this case, the remote servercan perform the one or more tasks, such as performing at least one ofSTEPS 2002-2006, and transmit an output to the computer system 100. Themethod 2100 can be performed by other components discussed herein andperform features and functionalities in conjunction with at least one ofFIGS. 1-20 , for example. The method 2100 can be performed inconjunction with, sequential, or prior to at least one of other methodsdiscussed herein, such as methods 400, 1100, 1500, etc.

Still referring to FIG. 21 , in further detail, the computer system 100can identify, from a PPG signal generated from a sequence of imagesacquired using a photodetector, various PPG pulse segments. The computersystem 100 can identify each of the PPG pulse segments using apredetermined (or predefined) sequence of signal feature types (STEP2002). The signal feature types can include at least a trough (e.g.,peak down), zero-crossing up, peak (e.g., peak up), and zero-crossingdown. The computer system 100 can identify the PPG pulse segments byclassifying portions of the PPG signal as the PPG pulse segments basedon the features in a predetermined ordered sequence of feature types.The computer system 100 can classify the portions of the PPG signal asthe PPG pulse segments similar to STEP 1908 in conjunction with FIG.20A.

The computer system 100 can determine, for each of the PPG pulsesegments, one or more respective pulse features determined from the PPGpulse segment (STEP 2004). The one or more respective pulse features caninclude at least the respective pulse duration for each PPG pulsesegment based on a length of time between a first signal feature and alast signal feature of the predefined sequence. The first signal featureand the last signal feature can be from the same family of feature types(e.g., from peak to peak, trough to trough, etc.). In some cases, thefirst signal feature and the last signal feature can be of differentfeature types, such as from an index of the start of a pulse to an indexof the end of the pulse. The index indicating the end of the pulse canbe different from (e.g., located before) the index for the start of thenext pulse. The index indicating the end of the pulse can be locatedbetween the diastolic max slope and the start of the next pulse.

In some embodiments, the respective pulse features can include theduration between two consecutive pulses or multiple pairs of consecutivepulses. For example, the computer system 100 can calculate a firstduration between the start of a first pulse to the start of a secondpulse. The computer system 100 can then calculate a second durationbetween the start of the second pulse to the start of a third pulse. Thecomputer system 100 can compare the first duration and the secondduration to determine a ratio or the variation between the durations oftwo pairs of consecutive pulses, in this case. The ratio (or durationratio) can of the durations between consecutive pulses (e.g., a pulsepair) can represent a quantitative relation between the duration of thepair of consecutive PPG pulse segments. The computer system 100 can usethe respective duration ratios for each of the pairs of consecutive PPGpulse segments to determine a reference duration ratio. The referenceduration ratio can include at least one of the maximum (or the minimum)of the respective duration ratios for the plurality of pairs ofconsecutive pulse segments. In some implementations, the referenceduration ratio can include a quantile (e.g., the magnitude orpercentage) of the respective duration ratios for the pairs ofconsecutive pulse segments.

Using the respective ratio for the pairs of consecutive PPG pulsesegments, the computer system 100 can detect if the user has acondition, such as an arrhythmia, based on the calculated ratio. Forexample, the computer system 100 can compare the reference durationratio (or one or more duration ratios of at least a pair of consecutivePPG pulse segments) to a threshold value. The computer system 100 candetermine if the arrhythmia condition should be raised based on thecomparison of the reference duration ratio to the threshold value. Inthis case, the threshold value can be θ_(cdr).

In some embodiments, the computer system 100 can determine a first pulseduration of a first PPG pulse segment and a second pulse duration of asecond PPG pulse segment to detect the condition of the user. Forexample, the first pulse duration can be a minimum pulse duration ofrespective pulse durations of the PPG pulse segments. The second pulseduration can be the maximum pulse duration of respective pulse durationsof the PPG pulse segments. In some cases, the first pulse duration canbe a first quartile of respective pulse durations of the PPG pulsesegments and the second pulse duration can be a second quantile ofrespective pulse durations of the PPG pulse segments. The computersystem 100 can determine a relative variation between the first pulseduration and the second pulse duration based on a comparison between themaximum and the minimum pulse duration from various PPG pulse segments.The computer system 100 can compare the relative variation between thefirst and second pulse durations to a threshold to determine whether toraise a condition (e.g., irregular heartbeat condition) based on arelative variation between the first pulse duration and the second pulseduration.

In some implementations, the computer system 100 can determine, for eachpair of consecutive pulse segments, various PPG pulse segments. Each PPGsegment or PPG pulse segment can include signal features that includethe predetermined sequence of signal feature types. For example, thepredetermined sequence of signal feature types can be in the order oftrough, zero-crossing up, peak (e.g., peak up), zero-crossing down, andtrough. In another example, the predetermined sequence of signal featuretypes can be in the order of peak, zero-crossing down, trough,zero-crossing up, and peak. The predetermined sequence can start withany other feature type.

The computer system 100 can compute the duration of individual pulses asdiscussed above with regard to subsection D.4. The computer system 100can use variation in pulse durations of consecutive pulses to detect oneor more conditions such as arrhythmia. High variation in pulse durationsof consecutive pulse can be indicative of an abnormal condition of theuser. The computer system 100 can compute a high ratio between the pulsedurations of pairs of consecutive pulses. Having a high pulse durationratio may suggest that something happened physiologically to the userand/or that the measurement may not be reliable. The computer system 100can compute the high pulse duration ratio between two successive pulsedurations as:

$\begin{matrix}{r_{i} = {{\max( {\frac{D_{i}}{D_{i + 1}},\frac{D_{i + 1}}{D_{i}}} )}.}} & (20)\end{matrix}$

The ratio r_(i) can represent the maximum or higher ratio of

$\frac{D_{i}}{D_{i + 1}}{and}{\frac{D_{i + 1}}{D_{i}}.}$

The ratio r_(i) can be equal to

$\frac{D_{i}}{D_{i + 1}}$

if D_(i) is greater than D_(i+1), or can be equal to

$\frac{D_{i + 1}}{D_{i}}$

if D_(i+1) is greater than D_(i). The computer system 100 can thencompute the maximal consecutive duration ratio as a high quantileq_(cdr) of the (r_(i))_(i≥0). The maximal consecutive duration ratio (orq_(cdr)-th quantile of the ratios r_(i)) can be denoted as r*. Thecomputer system 100 can detect one or more conditions based on whetherthe maximal consecutive duration ratio r* exceeds a threshold (e.g., aconsecutive duration ratio threshold θ_(cdr)). For example, the computersystem 100 may detect and raise (or report) a condition of Arrhythmia ifr* exceeds the consecutive duration ratio threshold θ_(cdr).

As in normal circumstances, the pulse duration of users may varythroughout time due to changes in the pulse rate, physiological state ofthe user, among other contributing factors to the user's blood volume.Further, with high variation in pulse duration, this may suggest thatthe user is not at rest (e.g., not still or moving while takingmeasurement). In this case, the measurement may not be reliable as itshould be, and the computer system 100 may notify the user to retry themeasurement or raise a condition indicating for the user to remain stillwhile taking the measurement, for example. The computer system 100 cancompute extremal durations between pulses, such as D^(min) for theminimum (or relatively short) pulse duration and D^(max) for the maximum(or relatively long) pulse duration from the PPG pulse segments. TheD^(min) and D^(max) may be computed as the q_(od) ^(th) and(1−q_(od))^(th) quantiles, respectively. The q_(od) can represent thequantile used to identify extremal pulse durations. The q_(od) can bepreset and configured, for example, to 1%, 2%, 3%, etc. In some cases,the q_(od) may be at least one of the highest or lowest values withinthe PPG pulse segments. The computer system 100 can determine themaximal duration ratio as:

$\begin{matrix}{R = {\frac{D^{\max}}{D^{\min}}.}} & (21)\end{matrix}$

Base on the maxima duration ratio R, the computer system 100 can detector raise (e.g., report) a condition indicative of irregular hear rate.For example, the computer system 100 can detect irregular heart rate ifR is greater than a corresponding threshold.

The computer system 100 can determine or detect one or more conditionsbased on the one or more respective pulse features of the various PPGpulse segments (STEP 2006). The one or more conditions can include atleast an arrhythmia condition or irregular heartbeat (or heart rate)condition. For example, the computer system 100 can determine or checkif the maximal consecutive duration ratio r* exceeds the thresholdθ_(cdr). If r*>θ_(cdr), the computer system 100 can detect or raise theerror condition of ARRHYTHMIA. In other words, the computer system 100can detect the arrhythmia condition based on the comparison of themaximal consecutive duration ratio to the threshold value of θ_(cdr).The computer system 100 can display the error condition on the displaydevice 112 for the user. In some cases, the computer system 100 cantransmit the error condition to an administrator for analysis. In someother cases, the computer system 100 can transmit the results and errorconditions to a machine learning model as a training data sample.

The computer system 100 can determine if R exceeds the maximal durationratio threshold θ_(or). For example, if R>θ_(or), the computer system100 can raise an error condition of IRREGULAR HEART RATE. In otherwords, the computer system 100 can detect and raise the irregularheartbeat condition based on a relative variation between the firstpulse duration and the second pulse duration. The relative variationbetween the first and second pulse durations can include or refer to aratio of the second pulse duration over the first pulse duration. Insome cases, the relative variation between the first and second pulsedurations can include or refer to a difference between the first pulseduration and the second pulse duration. In some other cases, therelative variation between the first and second pulse durations caninclude or refer to a normalized difference between the first pulseduration and the second pulse duration. The computer system 100 cannormalize the difference, for example, by dividing the difference with amaximum pulse duration, a minimum pulse duration, the median of thepulse durations, among other normalization techniques. Similar to theARRHYTHMIA error condition, the computer system 100 can display theerror condition to the user, transmit the error condition to theadministrator, or provide as an input to the machine learning modeleither for training or further processing to determine any falsepositive or false negative in the analysis of the pulse durations. Insome implementations, the computer system 100 can determine and raiseother conditions based on the duration or magnitude of the pulses. Theone or more conditions discussed above can be related to the physiologyof the user.

In some cases, the conditions can include or refer to the condition ofthe computer system 100, such as irregular data captured by thephotodetector. In this case, the computer system 100 may notify andrequest the user to reattempt the measurement. In some cases, since someusers may not be aware of some of the conditions (e.g., hand stabilitywhen taking measurements, etc.), the computer system 100 can notify someconditions as a hint to the user, such as to maintain the stability ofthe finger for more accurate measurement. Thus, based on the duration ofindividual pulses, the computer system 100 can determine one or morepotential conditions that the user may have using the PPG pulsesegments. An illustrative example of the pulse durations and the pulseduration ratios computed based on pairs of pulse durations can be shownin FIG. 22 .

In some implementations, the computer system 100 can identify ordetermine if there are a sufficient number or enough pulses forcomputing the pulse duration ratio for detecting at least the arrhythmiacondition or the irregular heart rate condition. The computer system 100can determine if there is either enough or insufficient number of pulsesbased on the number of extracted pulses, the duration of the measuredsignal, or a total number of captured frames. For example, the computersystem 100 can compare the total number of pulses to a threshold. If thetotal number of pulses is less than the threshold, the computer system100 can alert the user of an insufficient number of pulses condition(sometimes referred to as NOT ENOUGH PULSE condition). If there are asufficient number of pulses, the computer system 100 can proceed todetermine the pulse duration ratio, among other conditions ormeasurements, for example. In another example, the total number ofpulses can be associated with the total number of frames. In this case,the computer system 100 can compare the total number of frames to athreshold. If the total number of frames is less than the threshold, thecomputer system 100 can alert the user of an insufficient number ofpulses condition. If there are a sufficient number of frames, thecomputer system 100 can proceed to determine the pulse duration ratio,among other conditions or measurements. In some embodiments, thecomputer system 100 can compare the total duration of capturing the PPGsignal to a threshold. Similar to the previous example, if the totalduration satisfies the threshold, the computer system 100 can proceed toother steps in at least one of error checking (e.g., checking otherconditions) or performing a measurement. Otherwise, if the totalduration does not satisfy the threshold, the computer system 100 canalert the user of the insufficient number of pulses condition.

FIG. 22 shows plots illustrating an example PPG signal, variation inrespective pulse durations and variations in pulse duration ratios,according to example embodiments. The example illustration can include afirst subplot having a PPG signal, a second subplot having the pulsedurations, and a third subplot having the pulse duration ratios. Thefeatures and functionalities discussed herein, such as capturing the PPGsignal, generate PPG pulse segments, compute the pulse durations, ordetermine the pulse duration ratios can be performed by the computersystem 100 executing the application 114. In some implementations, thefeatures and functionalities discussed herein can be performed by aserver or a remote computing device, or the combination of the serverand the computer system 100. The computer system 100 can receive the PPGsignal including various PPG pulse segments. The PPG pulse segments canrepresent or include a pulse with a duration from the start of the pulseto either the end of the pulse or the beginning of the next pulse. Theend of the pulse and the beginning of the next pulse may be the sameindex within the PPG signal.

In the second subplot, the computer system 100 can determine the pulseduration for each of the PPG pulse segments from the PPG signal of thefirst subplot. The computer system 100 can calculate a pulse durationratio for each pair of pulse durations, as presented in the thirdsubplot. The computer system 100 can use the pulse duration ratio todetermine if the user has at least one of the conditions (e.g.,irregular heartbeat or arrhythmia). For example, through the measuredperiod of time, the computer system 100 can determine that the pulseduration is generally at 1 second. By commuting the ratio between afirst pulse duration and a second pulse duration of around 1 second, thepulse duration ratio can be 1:1 (e.g., pulse duration ratio of 1).However, at approximately 17 to 20 seconds, the computer system 100 candetect the variation of the user's pulse duration based on the computedratio, such as 1:0.7, 0.7:1.2, 1.2:2, and 2:1 for example. The ratioscan be represented as values, such as 1.43, 1.71, 1.67, and 2,respectively in this case. The computer system 100 can determine if theuser has arrhythmia, based on a comparison between at least one ratiogreater than a threshold θ_(cdr). For example, if θ_(cdr) is 1.7 and theprevious ratios are calculated, the computer system 100 can determinethat the user may have an arrhythmia, thereby raising or alerting thearrhythmia condition to the user. Otherwise, if the ratios from the PPGsignal are less than θ_(cdr), the computer system 100 can continuemeasuring the blood volume of the user. In some cases, the computersystem 100 can notify the user that no arrhythmia condition is detected.In some implementations, the computer system 100 may be configured toraise the arrhythmia condition after calculating a predetermined numberof ratios exceeding θ_(cdr), such as 2, 3, 5, 10, etc.

In another example, the computer system 100 can determine if the userhas an irregular heartbeat to raise the irregular heartbeat condition,based on at least one of a ratio, a difference, an average, or anormalization between the maximum duration and the minimum durationwithin the PPG signal. The computer system 100 can use other metrics,aggregation, differentiation, or measurement techniques to determine ifthe user has a condition. The computer system 100 can normalize themaximum and minimum durations of the PPG signal by dividing the resultof, for example, the difference between the maximum and the minimumdurations by at least one of the average, the difference, the median,the maximum, or the minimum durations. The computer system 100 cannormalize the maximum and minimum durations by other normalizationtechniques. Based on the two durations, the computer system 100 cancompute a ratio or a value for comparison with a threshold θ_(odr).Referring to the previous example, the maximum duration can be 2 secondsand the minimum duration can be 0.7 seconds. In this case, the ratiowould be 2.86. The computer system 100 can compare this result to thethreshold θ_(odr). If the ratio is above (or equal to) the thresholdθ_(odr), the computer system 100 can raise the irregular heartbeatcondition and alert the user of the condition. Otherwise, if the ratiois below the threshold θ_(odr), the computer system 100 may continuemeasuring the blood volume of the user or complete the measurementprocess.

D.5. Blood Perfusion Based Conditions

A good signal (e.g., PPG signal) can be or refer to a signal thatincludes much more information compared to noises. In other words, goodsignals can have a high signal-to-noise ratio (“SNR”) (e.g., more signalthan noise). To compute or analyze data from the signals, the computersystem 100 can determine if the signal is of good quality (e.g., largeSNR) or if there are more noises than the signal itself. The computersystem 100 can measure the perfusion (e.g., blood flow rate or thevolume of blood per unit time) from a body part of the user, such as thefinger. Perfusion measurement can revolve around having a PPG signalwith a large SNR, such that the signal has a great amplitude for thecomputer system 100 to compute or provide an accurate measurement.

The perfusion can be a measure of the amplitude of the signal. Thecomputer system 100 can determine the perfusion by using the log of thePPG signal as a measure of the absorbance to compare the amplitudes ofthe signal without being biased by the average value of the signal. Forexample, if the luminosity is 100 for a signal around 1000 (e.g., 1000values or points), the variation of this luminosity should not weighmore than a variation of 1 for a signal around 10. To determine theperfusion, the computer system 100 can capture a sequence of images tocalculate or determine the PPG signal used as input. The PPG signal canbe denoted as X, which can include a length of T The computer system 100can consider one or more parameters, such as the variation threshold,the window size, and the variation quantile when determining theperfusion. The computer system 100 can consider other parametersdiscussed herein to determine the perfusion of the signal.

The variation threshold can include or be a threshold for the variationmetrics of the PPG signal or values to be considered as having a goodamplitude. The variation threshold can be denoted as θ herein. Thevariation metric can include at least a value or a metric indicating thevariation (e.g., difference, standard deviation, average, among others)between the upper and lower values of individual PPG pulse segments, forexample. The variation metric can be based on the variation quantilediscussed herein. The computer system 100 can compare the variationmetric to 0, such as to determine if the PPG signal should be used forcalculating the blood volume, pulse rate, or other characteristics ofthe pulse. The window size can include or refer to the duration of awindow over which the variations are computed. The window size can bedenoted as n herein. The window size can be set to Is, 30 frames, 60frames, among other values or durations. In some cases, the window sizecan depend on the shutter speed of the camera. The window size can beconfigured by the administrator of the computer system 100. Thevariation quantile can include or be a quantile of the distribution ofamplitudes used for the variation metric. The variation quantile can bedenoted as q_(var) herein. The variation quantile can be configured to0.5 (e.g., a cutoff point at 50% or the median of the distribution ofamplitudes).

Based on the input signal and the parameters, the computer system 100can calculate or determine the perfusion of the PPG signal. Theperfusion can be denoted as P herein. The perfusion can be a singlevalue used to determine whether the PPG signal should be used tocalculate the characteristics of the user's blood such as bloodpressure, among other features of the blood flow. In some cases, theperfusion can include or be the variation metric for comparison with θ.With P, the computer system 100 can determine a boolean value associatedwith an error condition of COLD_FINGER (sometimes referred to generallyas cold finger condition). The computer system 100 can alert the coldfinger condition to the user via the UI of the application 114responsive to determining the error condition, for example.

Referring to FIG. 23 , a flowchart illustrating a method 2300 ofdetermining perfusion of a PPG signal is shown, according to exampleembodiments. The method 2300 can include obtaining a PPG signal (STEP2302). The method 2300 can include determining a logarithmic PPG signal(STEP 2304). The method 2300 can include determining an estimate ofblood perfusion using the logarithmic PPG signal (STEP 2306). The method2300 can include determining a blood perfusion condition based on theestimate of blood perfusion (STEP 2308). The method 2300 can beperformed by the computer system 100 or the corresponding processor 102,for example, upon executing the application 114. The method 2300 can beperformed by other components (e.g., memory 104, BUS 106, light source108, camera device 110, or display device 112) of the computer system100, in conjunction with FIG. 1 . The method 2300 can be performed by aremote device or a remote server. In some implementations, the method2300 can be performed by a computer system 100 and a remote server. Forexample, the computer system 100 can be in communication with the remoteserver to delegate one or more tasks to the remote server. In this case,the remote server can perform the one or more tasks, such as performingat least one of STEPS 2302-2308, and transmit an output to the computersystem 100. The method 2300 can be performed by other componentsdiscussed herein and perform features and functionalities in conjunctionwith at least one of FIGS. 1-22 , for example. The method 2300 can beperformed in conjunction with, sequential, or prior to at least one ofother methods discussed herein, such as methods 400, 1100, 1500, etc.

Still referring to FIG. 21 , in further detail, the computer system 100can obtaining, by a computing device, a PPG signal generated from asequence of images acquired using a photodetector (e.g., camera or RGBsensor 312) while a body part (e.g., a finger) is placed in proximityand in the field of view of the photodetector (STEP 2302). In somecases, the computer system 100 can obtain the PPG signal similarly to atleast STEP 1902 in conjunction with FIG. 20A. For example, the computersystem 100 can capture a sequence of images (e.g., a video) of a fingerfor a period of N time. The computer system 100 can obtain or generatethe PPG signal using a sequence of images with a single color channel,such as only green or only red. In some cases, the computer system 100can obtain the PPG signal using a sequence of images with multiple colorchannels (e.g., R, G, and/or B).

Prior to obtaining or generating the PPG signal, the computer system 100can process raw images captured using the photodetector. For example,the computer system 100 can generate a sequence of downsampled colorframes corresponding to the sequence of images by downsampling arespective color frame for each image of the sequence of images. Thecomputer system 100 can downsample the images to reduce the resolutionof the image for generating the PPG signal. By reducing the resolution(e.g., reduced to 5×5 pixels), the computer system 100 can decreaselatency and reduce resource consumption when processing the sequence ofdownsampled color frames. The downsampled color frames may include onlyone color channel, such as green or red. In some cases, the downsampledcolor frames may include multiple color channels.

Responsive to generating the sequence of downsampled color frames, thecomputer system 100 can identify, in each downsampled color frame of thesequence of downsampled color frames, a respective image blockrepresenting a central image region of the downsampled color frame andhaving a first size smaller than a second size of the downsampled colorframe. The respective image block can include or be referred to as aportion of the respective downsampled color frame. The central imageregion of the respective image block may be located at the center of thedownsampled color frame. For example, in a 5×5 pixels image (among othersizes), the computer system 100 can determine that the central imageregion is located at the inner 3×3 pixels of the downsampled colorframe. In another example, the computer system 100 can determine thatthe central image region is not located at the center or the innerportion of the downsampled color frame. Instead, the computer system 100can identify the center image region located edges of the downsampledcolor frame. The central image region may be of any size smaller thanthe downsampled color frame. For example, if the downsampled color frameis 10×10 pixels, the central image region can be 5×5 pixels, 5×9 pixels,7×4 pixels, among others.

The computer system 100 can generate a color intensity signal using therespective image blocks of the sequence of downsampled color frames. Forexample, the computer system 100 can identify, within the central imageregion of each respective image block, the color value or intensityvalue associated with image blocks. Based on the color value or theintensity value, the computer system 100 can generate a color intensitysignal having at least one of the color values or the intensity valueassociated with the color intensity signal. Accordingly, the computingdevice can generate the PPG signal using the color intensity signal todetermine a blood pressure value.

The computer system 100 can determine a logarithmic PPG signal bycomputing a logarithm of the PPG signal (STEP 2304). The logarithmic(sometimes referred to generally as a log) PPG signal can beproportional to the blood absorbance of light. For example, to estimatethe blood perfusion, the computer system 100 can use at least one of thefollowing metrics among other metrics for a PPG signal. The metric caninclude a transmitted light T∝X, which can be directly proportional tothe RGB values yielded or captured by the photodetector (e.g., thecamera or the RGB sensor 312). The metric can include an absorbance ofthe tissue, as in formula (22).

A=−log₁₀(T)∝log(X)  (22)

The absorbance value can be denoted as A. The A can be proportional tothe logarithm of the transmitted light based on formula (22) for A.

With A being proportional to the logarithm of the transmitted light,when T is subject to variations of the amplitude (e.g., with changes ofincoming light or with changes of the multiplicative factor), A can bemore robust (e.g., higher light absorbance by the tissue, which canreflect higher amplitudes for the PPG signal), with both the changes inincoming light or the multiplicative factor resulting only in variationsof the offset, for example. In other words, variation in T can increasethe robustness or absorbance level of A, with the changes in incominglight and multiplicative factor variating the offset of A. An example ofthe PPG absorbance A can be illustrated in part in FIG. 24A, forexample. The PPG absorbance A can refer to the logarithmic PPG signalcomputed from log₁₀ of T and X according to the formula discussed above.

The computing device can determine an estimate of blood perfusion in thebody part using the logarithmic PPG signal (STEP 2306). As an example,the body part can be the user's finger, among other body parts. Theestimate of blood perfusion can be based on the logarithmic PPG signal(or PPG absorbance A) calculated from the logarithm of the PPG signal.For example, the computer system 100 can determine the estimate of bloodperfusion based on the amplitude calculated from the logarithmic PPGsignal or the PPG absorbance A. In some cases, the estimate of bloodperfusion can include or refer to the amplitude of the PPG absorbance A.For example, the computer system 100 can determine the estimate of bloodperfusion by determining an envelope of the logarithmic PPG signal. Theenvelope can refer to the amplitude of the logarithmic PPG signal, e.g.,the difference between the upper value of A and the lower value of A foreach of the segments or portions of the logarithmic PPG signal.Accordingly, the computer system 100 can determine the estimate of bloodperfusion using the envelope of the logarithmic PPG signal. In otherwords, the envelope may be indicative of the amplitude of thelogarithmic PPG signal or at least a segment of the logarithmic PPGsignal.

In some implementations, determining the envelope of the logarithmic PPGsignal can refer to or include determining, for each time interval ofvarious time intervals of the logarithmic PPG signal, a respectivemaximum and a respective minimum of the logarithmic PPG signal. Forexample, the logarithmic PPG signal can include various time intervals,each representing an interval of a PPG absorbance A, such as a pulse ofA. For each time interval, the computer system 100 can determine theminimum (e.g., trough) and the maximum (e.g., upper peak) of A. Thecomputer system 100 can use the respective maximum and minimum todetermine the envelope of the logarithmic PPG signal. In some cases, thecomputer system 100 can use the respective maximum and minimum of thelogarithmic PPG signal to determine the estimate of blood perfusion inthe body part.

The computer system 100 can compute the amplitude of the logarithmic PPGsignal using the envelope of the signal. The computer system 100 candetermine the envelope, denoted as A_(lower) and A_(upper), by computingthe sliding maxima and sliding minima over a time period. In otherwords, the computer system 100 can use the maxima and the sliding minimaas the estimate blood perfusion, e.g., based on the amplitude of thesignal over time. The time period can refer to the time through thelogarithmic PPG signal. The A_(lower) and A_(upper) of the envelope canbe computed using formula (23).

$\begin{matrix}{{{A_{lower}(t)} = {\min\limits_{t^{\prime} \in {\lbrack{{t - \frac{dt}{2}},{t + \frac{dt}{2}}}\rbrack}}{A( t^{\prime} )}}},{{A_{upper}(t)} = {\min\limits_{t^{\prime} \in {\lbrack{{t - \frac{dt}{2}},{t + \frac{dt}{2}}}\rbrack}}{{A( t^{\prime} )}.}}}} & (23)\end{matrix}$

The computer system 100 can determine, for each of the time intervals, arespective local variation of the envelope of the logarithmic PPGsignal. The respective local variation can be equal to a differencebetween the respective maximum and the respective minimum of thelogarithmic PPG signal within the time interval. The local variation canbe referred to as, or used interchangeably with other descriptive terms,such as a local width or an amplitude of the envelope of the logarithmicPPG signal. Formula (24) can be used to compute the local variation.

Δ(t)=A _(upper)(t)−A _(lower)(t).  (24)

Accordingly, the computer system 100 can use the amplitude, which can beindicative of the blood perfusion, or the estimate of the amplitude todetermine the blood perfusion condition.

FIG. 24A illustrates an example of the logarithmic PPG signal (or PPGabsorbance A) and the respective envelope, according to exampleembodiments. For example, the illustration of FIG. 24A can include aplot of PPG absorbance A over time (e.g., timestamp in seconds). In thisexample, the range of A can be from 1.845 to 1.880 and the length oftime can be from 1 second to 35 seconds. Throughout the plot, thecomputer system 100 can compute and generate the logarithmic PPG signaldenoted generally as the PPG (signal) in the plotted line. The computersystem 100 can compute the envelope of the using the logarithmic PPGsignal within the time intervals. As shown, for each of the timeintervals, the envelope may vary in amplitude and offset. The computersystem 100 can compute the offset of the logarithmic PPG signal based onthe changes of incoming light or with changes of the multiplicativefactor, since A is proportional to the logarithm of the transmittedlight, for example. The width, girth, amplitude, or height of theenvelope can vary through each of the time intervals within thelogarithmic PPG signal. In particular, the computer system 100 can usethe envelope indicative of the difference between the upper A and thelower A to determine the amplitude. As such, the computer system 100 cancompute the amplitude for each of the time intervals of the logarithmicPPG signal. With the amplitude throughout the logarithmic PPG signal,the computer system 100 can generate a plot or a metric of thecumulative distribution of the amplitudes, indicating the number or acount for different magnitudes of amplitudes, as in the illustrativeexamples of FIGS. 24B-E.

In some implementations, determining the envelope of the logarithmic PPGsignal can include applying a low-pass filter to the logarithmic PPGsignal. The computer system 100 can apply the low-pass filter for eachof various time intervals of the logarithmic PPG signal. For example,for each time interval, the computer system 100 can apply a first lowpass filter to the logarithmic PPG signal, thereby discarding orignoring values beyond the maximum value of A for the respective timeinterval in the logarithmic PPG signal. The computer system 100 canflip, reverse, or compute the negative of the logarithmic PPG signal,such that the minimum value becomes the maximum value, and vice versa,in this case. The computer system 100 can apply a second low-passfilter, which can discard or ignore the values beyond the maximum valueof A (previously the minimum). The computer system 100 can revert thelogarithmic PPG signal to positive. Accordingly, the computer system 100can use the low-pass filter to determine the envelope of the logarithmicPPG signal. The computer system 100 can use other types of filters todetermine the envelope of the logarithmic PPG signal.

The computer system 100 can determine the estimate of the amplitude (orthe blood perfusion) of the logarithmic PPG signal using at least oneother technique. For example, the computer system 100 can determine theestimate of the blood perfusion as a predetermined quantile (e.g., 5%,10%, 90%, 95%, etc.) of the respective local variations of the envelopeof the logarithmic PPG signal within the time intervals. In someimplementations, the computer system 100 can determine the estimate ofthe blood perfusion as a median (e.g., quantile of 0.5 or 50%) of therespective local variations of the envelope of the logarithmic PPGsignal within the time intervals. In some implementations, the computersystem 100 can determine the estimate of the blood perfusion as anaverage or a mean of the respective local variations of the envelope ofthe logarithmic PPG signal within the time intervals. In someimplementations, the computer system 100 can determine the amplitude ofthe logarithmic PPG signal as an average or a mean of the respectivelocal variations of the envelope of the logarithmic PPG signal withinthe time intervals. In some implementations, the computer system 100 candetermine the estimate of the blood perfusion as a maximum (e.g.,maximum value) of the respective local variations of the envelope of thelogarithmic PPG signal within the time intervals. In someimplementations, the computer system 100 can determine the estimate ofthe blood perfusion as a minimum (e.g., minimum value) of the respectivelocal variations of the envelope of the logarithmic PPG signal withinthe time intervals.

After computing the amplitude of the logarithmic PPG signal, thecomputer system 100 can determine the distribution of the amplitudes ofdifferent time intervals within the logarithmic PPG signal. For example,the computer system 100 can distribute amplitudes calculated from eachof the time intervals of the logarithmic PPG signal in a graph havingthe cumulative distribution of the amplitudes. Examples of the graph canbe shown in FIGS. 24B-E. The distribution of the amplitudes may besimilar to a normal distribution, e.g., with some location and scale(e.g., x-axis and y-axis scaling), and some outliers. In this case, thelocation may represent the perfusion, which can be an amplitude withrespect to the cumulative distribution.

The computer system 100 can determine an estimate of blood perfusion inthe body part using respective local variations (e.g., amplitudes) ofthe envelope of the logarithmic PPG signal within the time intervals.Each local variation can correspond to the respective time interval ofthe logarithmic PPG signal. The computer system 100 can determine theblood perfusion condition (sometimes generally referred to as theperfusion indicated by the amplitude within the cumulative distributionof amplitudes) of the body part based on the estimate of the amplitudeof the logarithmic PPG signal. As an example, to obtain a good estimateof the perfusion without many outliers, the computer system 100 cancompute the median (e.g., a quantile of 0.5 or 50%) of the distributionto obtain the median perfusion. The computer system 100 can estimate theblood perfusion as:

$\begin{matrix}{P = {{\underset{t}{med}( {\Delta(t)} )}.}} & (25)\end{matrix}$

The “med” can denote the median of the distribution, which can be avalue such that 50% of the distribution is below or equal to it and 50%is above or equal to the respective value. The median can be an examplevalue or quantile used in this case. Instead of using the median, thecomputer system 100 can use other quantiles q_(var) to estimate theamplitude so that at least a proportion 1−q_(var) of the signal has atleast some amplitude P. Accordingly, the median can be a particular casewith q_(var)=0.5, and other quantiles (e.g., q_(var)=0.4, 0.45, 0.49,0.51, 0.55, 0.6, etc.) can be used to determine the perfusion based onthe distribution of amplitudes.

The computer system 100 can determine a blood perfusion condition of thebody part based on the estimate of blood perfusion (STEP 2308). Theblood perfusion condition can include or indicate a low blood perfusioncondition indicative of a cold body part. Based on the estimate of theblood perfusion (e.g., the estimate of the amplitude of the cumulativedistribution of amplitudes), the computer system 100 determine if theblood perfusion condition, such as the user having low blood perfusion,should be raised. Examples of acceptable or unacceptable perfusionsbased on logarithmic PPG signals can be shown in FIGS. 24B-E.

For example, the computer system 100 can compare the estimate of theamplitude or the estimate of the blood perfusion (e.g., the value of P)of the logarithmic PPG signal to a threshold value θ to determine if theperfusion of the user is low or normal. The computer system 100 candetermine the blood perfusion condition of the body part (e.g., whetherthe user has a low perfusion or a normal perfusion) based on thecomparison of the estimate of the amplitude of the logarithmic PPGsignal to the threshold value. If P≤θ based on the comparison, thecomputer system 100 can determine that the perfusion is low. Otherwise,if P>θ, the computing device can determine that the perfusion is normal.In some implementations, if P is above a second threshold configured bythe administrator, different from θ, the computer system 100 maydetermine that the perfusion is high, which may raise a different bloodperfusion condition (e.g., high blood perfusion condition indicative ofan abnormally hot body part).

The threshold θ can be configured by the administrator. In someimplementations, the threshold θ can be determined by a machine learningmodel. For example, the computer system 100 can feed various samples ofPPG signals with PPG absorbance A having different amplitudes. Thesesamples of PPG signals can be marked or labeled indicating one or morefeatures of the PPG signals, such as the peak up, peak down,zero-crossing up, zero-crossing down, duration of the PPG pulsesegments, among others. The machine learning model can attempt tocompute these features from the sample PPG signals having the respectivePPG absorbance A with different amplitudes. Based on the performance(e.g., the accuracy) of feature extractions, the machine learning modelcan determine, provide, or indicate a threshold θ predetermined orpreset for the computer system 100, such that perfusion above thethreshold θ can be accepted. Hence, the computer system 100 can providean accurate measurement of the blood characteristics using acceptableperfusion calculated from the logarithmic PPG signal.

The computer system 100 can provide, responsive to determining the bloodperfusion condition, an alert to the user placing the finger on thelens. The alert can indicate a cold body part or that the body part onthe lens is cold, which may produce a lower accurate measurement. Insome implementations, the alert can include one or more instructions forwarming up the body part. Accordingly, the computer system 100 caninform or alert the user to retry the measurement with increasedperfusion (e.g., increase the temperature of the body part used formeasurement), for example. By performing the features andfunctionalities discussed above, and using the estimate of bloodperfusion to determine the blood perfusion condition, the computersystem 100 can capture good quality PPG signals with high SNR forincreasing the accuracy of blood volume measurement for identifying anyconditions of the user.

Referring to generally to FIGS. 24B-E, example illustrations of thelogarithmic PPG signal, the envelope of the logarithmic PPG signal, andtheir corresponding PPG amplitude distribution are shown, according toinventive concepts of this disclosure. The example illustrations caninclude plots (e.g., graphs or subplots) 2402-2416, where plots 2402,2406, 2410, and 2414 corresponds to plots 2404, 2408, 2412, and 2416,respectively. The features, functionalities, operations, or techniquesperformed in relation to the example illustrations can be performed bythe computer system 100 executing the application 114, a remote server,among other components herein. In some cases, the features andfunctionalities can be performed by the combination of the computersystem 100 and the remote server. The features and functionalitiesdiscussed herein can be performed in conjunction with methods andoperations discussed in at least one of FIGS. 1-22G, for example.

The computer system 100 can compute the logarithmic PPG signals forplots 2402, 2406, 2410 and 2414 based on the respective PPG signals.Using the logarithmic PPG signals, the computer system 100 can computethe respective envelopes as shown in the plots 2402, 2406, 2410 and1414. The computer system 100 can compute the respective amplitudes ofan envelope for each of the time intervals within the logarithmic PPGsignal. With the respective amplitudes for each logarithmic PPG signal,the computer system 100 can determine the distribution of theamplitudes, as presented in plots 2402, 2408, 2412 and 2416, forexample.

The computer system 100 can determine the perfusion of the distributionof amplitude based on a predetermined quantile indicative of theperfusion. In the example illustrations, the quantile can be configuredto 0.5 or 50%. With a quantile of 0.5, the computer system 100 candetermine an estimate of the amplitude or a value at the median or 50%of where all amplitudes fall within the plot. In some otherimplementations, the computer system 100 can be configured to determinethe perfusion at a different quantile, such as the average/mean of allamplitudes, a quantile of 0.4, 0.6, 0.55, among other quantiles. In someembodiments, the computer system 100 can determine the median perfusionfor plot 2404 as approximately 0.0007 amplitude, for plot 2408 asapproximately 0.0026, for plot 2412 approximately 0.0042, and for plot2416 approximately 0.009.

The computer system 100 can compare the perfusion to a threshold todetermine if the condition (e.g., low perfusion condition, sometimesreferred to as cold finger condition) should be raised. The thresholdmay be configured by the administrator of the computer system 100 ordetermined by a machine learning model trained using historical PPGsignals, historical logarithmic PPG signals, among other samples.Further from the previous examples, the computer system 100 can comparethe determined perfusion to a threshold of 0.004. The threshold can bepresented in the plots 2404, 2408, 2412 and 2416 as a perpendicular lineat the x-axis value of 0.004. Perfusions less than or equal to thethreshold can be considered as low perfusion, which may trigger a lowperfusion condition. Perfusions higher than the threshold can beconsidered normal perfusion.

Based on the calculated median perfusions, the computer system 100 candetermine that the median perfusions of plots 2404 and 2408 are lowerthan the threshold and that the median perfusions of plots 2412 and 2416are higher than the threshold. Accordingly, the computer system 100 canalert or notify the user of the low perfusion condition for logarithmicPPG signals of FIGS. 24B-C. As for FIGS. 24D-E, the computer system 100can proceed to measure the results of the blood volume or otherconditions of the user using the respective PPG signal captured from thebody part (e.g., the finger) of the user.

In some implementations, the computer system 100 can compare theperfusion to a second threshold representing the maximum allowedperfusion (not shown). For example, the computer system 100 candetermine that the perfusion at the preconfigured quantile (e.g.,median, average, etc.) higher than the second threshold can indicate ahigh perfusion condition or that the finger is too hot, in somecircumstances. Therefore, similar to the previous example, the computersystem 100 can alert the user of high perfusion conditions if theestimate of blood perfusion exceeds or equal to the second threshold.Otherwise, the computer system 100 can proceed to determine otherconditions or to measuring the blood volume results for the user if theperfusion is below the second threshold.

D.6. Saturation Based Conditions

For a PPG signal to be considered as a good PPG signal, the signalshould be in the linear range of the camera or exhibit linearcharacteristics between the incident physical light and measured light.Therefore, the saturation of the images should be balanced, such as nottoo dark and not too bright. The features, functionalities, operations,or techniques discussed herein can be performed by the computer system100, a remote server, or the combination of the computer system 100 andthe remote server. As an example, the computer system 100 can achieve atmaking the digitalized signal, which is a transformation of the physicalsignal, as close to a linear transformation as possible. The computersystem 100 can determine, for each image frame of a sequence of imageframes, if pixel values of the image are within the linear range orexhibit non-linear characteristics.

The computer system 100 can capture and downsample images of the bodypart (e.g., a finger of the user) from the camera. The downsampledimages can include or be preconfigured to a size of 5×5 pixels, forexample. The computer system 100 can downsample the images to any othersize based on the configuration for downsampling images. To determinethe level of saturation for each of the image frames, the computersystem 100 can be configured with one or more thresholds, such as a darksaturation threshold or a bright saturation threshold. The computersystem 100 can compare the macropixel data or the pixel color values ofeach image to the thresholds. The dark saturation threshold and thebright saturation threshold can be preconfigured by anoperator/administrator or determined by a machine learning modelanalyzing the threshold values (e.g., upper and lower values) for thecolor values to be within the linear range, such as in the measuredlight to the incident physical light spectrum. The dark saturationthreshold can be denoted as θ_(min) ^(c), where c can denote the colorchannel, e.g., θ_(min) ^(green), θ_(min) ^(red). By comparing themacropixel data to θ_(min) ^(c), the computer system 100 can determineif the macropixel data is too low for being in the linear range.

The θ_(min) ^(c) can be predetermined or preset to a value, such as 5,10, 15, 20, among other values lower than the bright saturationthreshold. In this case, the values for the macropixel data and thethresholds may range from 0 to 255 for 8-bits colors, for example. Inother cases, the values for the macropixel data and the threshold mayhave a higher or lower range, depending on the bit size of the colorspectrum or color values captured by the camera. The thresholds (e.g.,dark saturation threshold or bright saturation threshold) can be set foreach color channel or for a combination of multiple color channels.

The bright saturation threshold can be denoted as θ_(max) ^(c), whichcan be the threshold for the maximum macropixel data. The computersystem 100 can use θ_(max) ^(c) to determine if the macropixel data istoo high for being in the linear range. The θ_(max) ^(c) can be athreshold for each of the color channel, e.g., θ_(max) ^(green), θ_(max)^(red). The θ_(max) ^(c) can be preconfigured or preset to a value, suchas 240, 245, 250, among other values. In this case, the maximum valuemay not exceed 255. Further, to determine if a saturation conditionshould be raised, the computer system 100 can determine if a thresholdor a total number of maximal incorrect frames is exceeded based on asequence of images compared to θ_(min) ^(c) and θ_(max) ^(c). Themaximal incorrect frames can be denoted as Nsat. The computer system 100can compare the total number of frames that do not satisfy θ_(min) ^(c)or θ_(max) ^(c), to Nsat to determine if there are too many incorrectframes. The Nsat can be preconfigured to any number of image frames,such as 30, 60, 90, 120, among other numbers of image frames. Whencomparing the total number of frames that do not satisfy θ_(min) ^(c) orθ_(max) ^(c) to Nsat, the total number of frames may be consecutiveimage frames or non-consecutive image frames within a sequence ofimages. Accordingly, with the parameters discussed above, including thethresholds and Nsat, the computer system 100 can determine if an errorcondition (e.g., isDarkSaturating or isBrightSaturating) should beraised.

In further detail, the computer system 100 can capture images of theuser's finger using a camera or a photodetector. The camera can be adevice that transforms a continuous physical value (e.g., lightintensity) into a discrete numerical value. Referring back to FIG. 11 ,a plot of a transfer function between incident physical light (e.g.,light intensity or value emitted by the light source 108 or the lightgoing into the camera) and the measured light (e.g., light intensity orvalue captured by the camera) can be shown. The data of the plot can bepresented on a logarithmic scale (e.g., increment of the power of 10).The camera can transform the continuous physical value into a discretenumerical value from 0 to 255, however, the light emitted by the lightsource 108 can exceed beyond an intensity or the value of 255.

Still referring to FIG. 11 as an illustrative example, the plot caninclude three lines noted as no quantization, pixels with noise andquantization, and pixels without noise and quantization. First, with noquantization line, no quantization nor transformation has occurred inthis case, which the transfer function can be the identity function.Hence, the no quantization line can be the reference line for an idealcase with linear characteristics at any range presented in the plot. Ina second case, if individual pixels are taken without any noise norbinning (e.g., no averaging multiple pixels or no downsampling), thetransfer function would have a stair-like representation (e.g.,non-linear characteristics). In this second case, the transfer functioncan be very rough at the low incident physical light and measured lightvalues (e.g., labeled as artifacts range due to quantization) andthresholded at the higher values (e.g., beyond the 250 value or othervalues in the thresholding range).

In a third case, if pixels are subject to some noise and averagedlocally (e.g., downsampled or averaged between multiple pixels), whilethe transfer function may be represented with stair-like characteristicslow values, the transfer function can be smoother than in the secondcase. For example, the higher the incident intensities (e.g., higherincident physical light value or measured light value), the pixel valuesof the pixels can be more smoothly until the thresholded value or range.In this third case, the medium range (e.g., labeled as a linear range)of pixel values can better represent or achieve the linearitycharacteristic of the identity transfer function. The linear range canbe from 100 to 150, 90 to 160, among other ranges which can bepredetermined based on analysis of historical transfer functions.

By downsampling the image for determining the saturation, the computersystem 100 can evaluate the saturation of the image based on the thirdcase discussed above, such as with noisy individual pixels andaggregation of batches of pixels into a single macro pixel bydownsampling the image. The computer system 100 can limit the spread ofvalues to an acceptable range in the approximatively linear portion ofthe transfer function, e.g., within the linear range, as shown in FIG.11 . To determine if the pixel values are in the acceptable range orwithin the linear range, the computer system 100 can classify one ormore images into multiple non-disjoint and non-exhaustive categories,such as too bright, too dark, a combination of too bright and too dark,or normal light intensity.

For example, the computing device can determine if the color value, suchas red or green is less than θ_(min) ^(c) of the respective colorchannel. If at least one macro-pixels is too dark (e.g., red<θ_(min)^(red) or green<θ_(min) ^(green)), the computer system 100 can flag theimage as isDarkSaturating or raise the dark saturation condition (e.g.,DARK_SATURATION) indicating that at least a portion of the image is toodark. In another example, the computer system 100 can determine if thecolor value is greater than θ_(max) ^(c), which indicates that at leasta portion of the image is too bright. If at least one macro-pixels istoo bright (e.g., red>θ_(max) ^(red) or green>θ_(max) ^(green)), thecomputer system 100 can mark or flag the image as isBrightSaturating orraise the light saturation condition (e.g., LIGHT_SATURATION) indicatingthat at least a portion of the image is too bright. By comparing thecolor values to the thresholds, the computing device can determine aboolean value corresponding to the respective error condition, such asto raise or not to raise the error condition. The threshold can be setto θ_(min) ^(green)=θ_(min) ^(red)=10 and θ_(max) ^(green)=θ_(max)^(red)=245, for example. The threshold can be set to other values, suchas a minimum of 100 and a maximum of 150, a minimum of 140 and a maximumof 200, etc.

Subsequent to raising at least one of the conditions, the computersystem 100 may alert the user of the error condition(s). For example,the computer system 100 can alert the user of the light_saturation orthe dark_saturation condition based on the determined condition. Byalerting the user, the user can maneuver the finger to correct oraddress the alerted condition. In some cases, the computer system 100can provide a hint to the user, such as to better cover the lens withthe finger if the image is too bright, increase the light intensity, orcover a portion of the finger on the light source 108, if the image istoo dark. In some implementations, the computer system 100 can aggregatethe flagged or marked images within a sequence of images to determinethe total number of images that are too bright or too dark. The computersystem 100 can compare the total number of images marked with at leastone error condition to Nsat. If the total number of flagged imagesexceeds the threshold of Nsat, the computer system 100 can alert theuser of at least one of the respective error conditions. The computersystem 100 can alert the user with the error condition that arepresented the most or with any error condition that is flagged. Thetotal number of flagged images can consecutive flagged images or may notbe consecutive images (e.g., total number of flagged images within thesingle sequence of images).

By determining whether the pixel values of an image are within thelinear range, the computer system 100 can reduce the likelihood ofartifacts due to the camera hardware. Further, by alerting the user ofthe error conditions or hinting the user of how to resolve the errorconditions (e.g., readjust the placement of the finger), the computersystem 100 can obtain images with pixel values within the linear range.If no error condition is detected in this case, the computer system 100can determine if other error conditions are triggered or proceed tomeasure the PPG of the user. Thus, the computer system 100 can increasethe accuracy of measuring the PPG of the user. The features andfunctionalities discussed herein can be performed independently,concurrently, or as a part of other features or functionalitiesdescribed, for example, in other methods or other error detectiontechniques.

D.7. Accelerometer Based Conditions

To obtain a good PPG signal, the user or at least a body part (e.g., afinger) of the user present on the photodetector should be still. Inother words, the user's finger should not move during the measurementprocess. As such, an accelerometer can be used to capture the movementor measure the stillness of the user. The features, functionalities,operations, or techniques discussed herein can be performed by thecomputer system 100, a remote server, or the combination of the computersystem 100 and the remote server. For example, the computer system 100executing the application 114 can detect the movement of the user usingan accelerometer which may be built-in, adapted, or connected to thecomputer system 100. In this case, the acceleration of the computersystem 100 can reflect or correspond to the acceleration or changes inthe movement of the user. The acceleration can be measured in meter persecond squared, inches per second squared, or other units for measuringacceleration.

The computer system 100 can capture one or more acceleration amplitudesassociated with the time of capturing a sequence of images. Theacceleration amplitude can be a metric (sometimes referred to as anacceleration metric) that measures the overall movement amplitude basedon variations around acceleration at rest. The acceleration metric canbe used to estimate the average movement intensity. The accelerationamplitude can increase (or decrease) proportionally to the changes inthe user's position, such that extensive position changes can greatlyincrease the metric and short position changes may not increase themetric at least by a noticeable amount (e.g., short position changeshave a small impact towards the average movement intensity).

In further detail, the computer system 100 can capture an accelerationsignal from an accelerometer coupled to or in electrical communicationwith the computer system 100. The acceleration signal can be denoted asX, which can have a length of T. The acceleration signal or data can beplotted in a graph including or composed of three components (e.g.,three individual signals) corresponding to the respective x, y, and zaxes from the device. The three components can be individualacceleration signals measuring the acceleration of the computer system100 or the user on the respective axis.

When capturing the acceleration, the acceleration data can be pollutedor affected by the earth's gravity or gravitational acceleration.Ignoring terms (e.g., ignoring variables or expression from thecalculation) created by the variation of the orientation of the computersystem 100 (e.g., ignoring the angular acceleration), formula (26) canbe presented for use by the computer system 100 to determine themeasured acceleration for each axis.

$\begin{matrix}\{ {\begin{matrix}{{a_{x}^{measured}(t)} = {{{a^{device}(t)} \cdot {x(t)}} - {g \cdot {x(t)}}}} \\{{a_{y}^{measured}(t)} = {{{a^{device}(t)} \cdot {y(t)}} - {g \cdot {y(t)}}}} \\{{a_{z}^{measured}(t)} = {{{a^{device}(t)} \cdot {z(t)}} - {g \cdot {z(t)}}}}\end{matrix}.}  & (26)\end{matrix}$

In this case, the formula can include the gravity variable g for theacceleration formula of each axis. The constant gravity g can bemodulated by the direction of the computer system 100 (e.g., g·x(t),g·y(t), or g·z(t)). To remove g, the computer system 100 can separate gfrom the direction of the computer system 100 by constructing or usingformula (27).

$\begin{matrix}{{a^{measured}}^{2} = {{( {{{a^{device}(t)} \cdot {x(t)}} + {g \cdot {x(t)}}} )^{2} + ( {{{a^{device}(t)} \cdot {y(t)}} + {g \cdot {y(t)}}} )^{2} + ( {{{a^{device}(t)} \cdot {z(t)}} + {g \cdot {z(t)}}} )^{2}} = {{a^{device}}^{2} + {g}^{2} + {2 \cdot a^{device} \cdot {g.}}}}} & (27)\end{matrix}$

With the above formula separating g from the direction of the computersystem 100, the computer system 100 can use a high-pass filter on∥a^(measured)∥² to remove the component ∥g∥² in formula (27). Hence, thecomputer system 100 can allow for values proportional to theacceleration of the device ∥a^(device)∥ to be considered whencalculating for ∥a^(measured)∥², for example. The high-pass filter canbe set at a cutoff frequency of 2.5 Hz, for example. In some cases, thecutoff can be set to a different frequency, such as 3 Hz, 4 Hz, 5 Hz,etc. The cutoff frequency can be denoted as f_(c). Accordingly, by usingthe high-pass filter, the computer system 100 can filter out the gravityacceleration ∥g∥². The high-passed filtered signal with a cutofffrequency f_(c) can be denoted as ã.

With the above formula removed of ∥g∥², the computer system 100 candetermine a by using the acceleration of the device to calculate for ã.The computer system 100 can use at least one of the accelerations from arespective axis, multiple accelerations from the respective axes, or anaveraged acceleration between the axes, for example. The high-passedfiltered signal a can be computed for various portions and timeframe ofthe acceleration signal. Thus, the computer system 100 can compute orobtain various variations of amplitudes (e.g., envelope) of theacceleration signal distributed across a time period.

After computing the filtered acceleration ã, the computer system 100 cancompute the median amplitude of the filtered acceleration ã based on thepredetermined quantile of the distribution of acceleration values(sometimes referred to as an acceleration amplitude quantile). Theacceleration amplitude quantile can be denoted as q and used fordetermining the variation metric or the amplitude of the distribution ofamplitude values in the acceleration signal. In this case, the quantilecan be preconfigured to 0.5 or 50%, which represents the medianamplitude across the distribution of amplitude values. Formula (28) canbe used to determine the acceleration amplitude.

$\begin{matrix}{A = {\underset{t}{med}{{❘{\overset{\sim}{a}(t)}❘}.}}} & (28)\end{matrix}$

The “med” can denote the median of the distribution, e.g., the valuesuch that 50% of the distribution is below or equal to it and 50% isabove or equal to the median value. In this case, the median amplitudecan be the acceleration amplitude denoted as A representing a value ofthe acceleration amplitude of the signal. The median can be a particularcase with q=0.5. The computer system 100 can use other quantiles (e.g.,the average, the mode, among other values, such as 0.45, 0.48, 0.52,0.55, etc.) to estimate the amplitude A at q of at least a portion ofthe acceleration signal.

Responsive to or immediately after determining the amplitude A, thecomputer system 100 can compare A to a threshold value to determine ifthe movement of the device or the user is low or high. The amplitude Acan be in the same unit as the acceleration, such as meter per secondsquared. The threshold can refer to an acceleration amplitude threshold,denoted as θ. The computer system 100 can use the threshold θ todetermine if the user is still or in motion (e.g., extensive movement ornot stable). The threshold can be predetermined by the administrator ora machine learning model. For example, the machine learning model can betrained using sample data. The sample data can include PPG signal dataand historical acceleration data having various acceleration amplitudes.The machine learning model can use the sample data with variousacceleration amplitudes as references to determine differentacceleration amplitudes that yield good or poor results for identifying,extracting, or analyzing features from the PPG signal, for example. Inanother example, the threshold θ can be preconfigured to 0.05, which byhaving an amplitude below this value can be considered to yield goodresults when measuring the PPG of the user. Other values can be used asthe threshold θ, such as 0.04, 0.045, 0.055, 0.06, etc.

For example, if A≥θ, the computer system 100 can determine that theboolean value for “isMovementDetected” (sometimes referred to asMOVEMENT condition) is true (e.g., a value of 1). In this case, thecomputer system 100 can raise the movement condition indicating that theuser is not stable or moving during the measurement. By raising themovement condition, the computer system 100 can alert the user of theerror condition. In some cases, the computer system 100 can provide theuser with a hint to avoid the error condition in the next measurementattempt. The hint can include notifying the user to be more stable orstay still, avoid movement, place the device (e.g., the computer system100) on a flat/stable surface, among other suggestions.

In some cases, the computer system 100 can determine that A<θ. In thiscase, the computer system 100 can determine if any other errorconditions should be raised or proceed to measure the PPG of the user.Accordingly, the computer system 100 can determine how still the user isbased on the acceleration data and the amplitude of the accelerationbased on the quantile of the distributed acceleration values. In someimplementations, the computer system 100 can extract the envelope of theacceleration signals, similar to extracting the envelope for PPGabsorbance. By extracting the envelope of the acceleration signals, thecomputer system 100 can determine at least a region or a portion of themeasurement time when the user is still. Thus, if the user is not stillor moving at times during the measurement, the computer system 100 canuse at least the region when the user is still to measure the PPG of theuser, such that accurate PPG measurements can be obtained at least forthe respective region.

D.8. Detection of Data Error Conditions Using a Color-Based Penalty

In subsection D.2, a cross-correlation based approach is described fordetecting or estimating a relative position of a body part (or finger)of user based on an acquired sequence of images. The computer system 100can use the estimated or the detected relative position of the body part(or finger) to provide feedback to the user. The feedback allows theuser to properly place their body part (or finger) against the lens ofthe camera or photodetector to ensure a sequence of images of a qualitythat allows accurate and reliable detection or estimation of the bloodpressure (or other vital signs) of the user.

In this subsection a color based approach for determining the relativeposition of the body part or finger with respect to a desired position(e.g., relative to the lens of the camera or the photodetector) isdescribed. The computer system 100 can compute or determine penaltyscores for various pixel positions based on color differences betweencolor values of pixels corresponding to the pixel positions and adesired or predefined color value representing a color of light emittedor reflected from the body part towards the photodetector.

A finger or other body part well placed on the lens is a key factor forgetting or generating a good quality PPG signal to allow for an accurateand/or reliable measurement of blood pressure. To that end, methods andsystems described herein include processing a captured image sequence inorder to help the user properly placing or positioning their finger orbody part on the lens of the camera or photodetector. A finger helperapproach can be based on the idea that when the finger (or body part) iswell positioned, light goes through (and gets reflected from) the fleshgiving it an orange color. When some of the light does not go throughthe finger it most likely keeps its original color (e.g., white).Accordingly, the computer system 110 can use color as an indicator ofwell or proper positioning or placement of the finger or other bodypart.

Referring now to FIG. 25A, a flowchart illustrating a method 2500 ofdetecting placement of a body part (e.g., a finger) facing aphotodetector is shown, according to example embodiments. In a briefoverview, the method 2500 can include obtaining a sequence of imagesacquired by the photodetector responsive to light emitted by orreflected from the body part (STEP 2502). The method 2500 can includedetermining, for each pixel position of a plurality of pixel positionsassociated with the sequence of images, a respective penalty scoreindicative of a similarity between a color value of a pixel of the pixelposition and a desired color value (STEP 2504). The desired color valuecan represent a color property of light emitted by or reflected frombody parts when placed opposite to the photodetector. The method 2500can include determining, using penalty scores of the plurality of pixelpositions, a relative position of the body part of the user with respectto a desired position (2506). The method 2500 can be performed by thecomputer system 100 or a computing device thereof. The method 2500 canbe performed, or executed, by the computing device hosting thephotodetector or camera 110 or another computing device (e.g., a remoteserver, desktop or laptop, among others) that is communicatively coupledto the device capturing or recording the sequence of images.

The method 2500 can include the computer system 100 obtaining a sequenceof images acquired by the photodetector responsive to light emitted byor reflected from the body part (STEP 2502). The photodetector or camera110 can acquire the sequence of images as discussed above, for example,with regard to step 402 of FIG. 4 , step 1602 of FIG. 16 or step 1802 ofFIG. 18 . The computing device hosting the photodetector 110 or anothercomputing device can obtain the acquired or recorded sequence of images.The computer system 100 can process image frames of the sequence ofimages sequentially as they are received.

In some implementations, the method 2500 can further include thecomputer system 100 downsampling each image frame of the sequence ofimages to generate a corresponding sequence of downsampled images, e.g.,as discussed above with regard to steps 404, 1604 and/or 1804 of FIGS.4, 16 and 18 . Downsampling the sequence of images can lead tosignificant reduction in the method 2500.

The method 2500 can include determining, for each pixel position of aplurality of pixel positions associated with the sequence of images, arespective penalty score indicative of a similarity between a colorvalue of a pixel of the pixel position and a desired color value (STEP2504). The desired color value can represent a color property of lightemitted by or reflected from body parts when placed opposite to thephotodetector. In some implementations, the color property can be (orcan include) the orange color.

The computer system 100 can generate for each image of the sequence ofimages a corresponding penalty matrix. The penalty matrix can have asize equal to the size of each image (or color frame) of the sequence ofimages or the size each downsampled image (or downsampled color frames).For example, if each downsampled image frame has a size of (d_(x),d_(y),3) or (d_(x), d_(y)), then each penalty matrix can have a size(d_(x), d_(y)). The penalty matrix can be denoted as Δ, and can includepenalty scores for pixels (or pixel positions) of a correspondingdownsampled image frame (or corresponding image frame of the sequence ofimages). For instance, each pixel in the downsampled image frame canhave a corresponding penalty score in the corresponding penalty matrixΔ.

In some implementations, with a corrected white balance applied by theapplication 114 (or other module associated with the camera 110) thewhite color may be detected by the camera or photodetector 110 as green.Referring to FIG. 25B, raw images captured while a user moves his fingeracross the lens of the camera are shown. Some exposure adjustmentsoccurred during the process. The green pixels (e.g., portions indicatedvia the dashed circles) correspond to portions of the image where thefinger is not properly stuck to the lens. Light bounces off (or isreflected by) the finger, without going through the finger, and gets inthe lens to the photodetector. The images in FIG. 25B provide anillustration that color can be a good indicator of whether or not thefinger is well positioned or placed against the lens.

In the RGB color space, determining the color of the finger (or otherbody part) may not be straightforward. For instance, the finger color(whether as a color value or a color range) cannot be specified ordetermined using a single color channel among the red, green and bluechannel. In contrast, the hue, saturation, luminance (HSL) color spaceallows for determining or specifying the finger (or other body part)color. In general, the HSL color space defines colors more naturallythan other color spaces. The hue channel carries or reflects the colorof captured light. The saturation channel, as the name suggests, depictssaturation or color purity. The luminance channel represents brightness.

FIGS. 25C-E show the images of FIG. 25B in the HSL space. FIG. 25C showsthe hue components of the images of FIG. 25B. The orange colorrepresents light that was absorbed by the finger (or other body part)and then reflected back to be received by the photodetector. Pixelswhose color drifts towards the green color are indicative of regionswhere light was not properly absorbed by the finger, and suggest thatthe finger (or other body part) should be moved toward that specificregion.

FIG. 25D shows the saturation components of the images of FIG. 25B. Theimages of FIG. 25D show that some of the pixels are saturating when toomuch light comes into them. FIG. 25E show the luminance components ofthe images of FIG. 25B. The luminance components depict brightness ateach. It is to be noted that in most of the images of FIG. 25E, thecentral region of the images is generally the one with highest luminanceor brightness. The images in FIGS. 25B-E correspond to a downsampledsequence of images.

Referring back to FIG. 25A, the method 2500 can further include thecomputer system 100 transforming the sequence of images or thecorresponding downsampled sequence of images to the HSL color space. Forinstance, the computer system 100 can convert the RGB color channels tohue, saturation and luminance color channels. The transformation isnonlinear, and the color of each pixel is depicted in the correspondinghue value of the pixel. To perform the transformation, the computersystem 100 can define or determine the Chroma C in terms of the extremalRGB channel values. Specifically, the computer system 100 can determinem as the minimum color value among the R, G and B color channels, e.g.,m=min (R,G,B), and determine Mas the maximum color value among the R, Gand B color channels, e.g., M=max (R,G,B). The computer system 100 canthen determine the Chroma C as C=M−m. The Chroma can be viewed asrepresenting an absolute amount of pure color in all colors.

Referring now to FIG. 25F, a diagram illustrating the chromatic circleis shown. The chromatic circle is designed so that different colors areassigned different degrees in the circle. The computer system 100 candetermine the hue value H for each pixel as an angle of a correspondingcolor on the chromatic circle:

$\begin{matrix}{H = {6{0^{o}.\{ {\begin{matrix}{{0{if}\ C} = 0} \\{{\frac{G - B}{C}{mod}\ 6{if}\ M} = R} \\{{\frac{B - R}{C} + {2\ {if}\ M}} = G} \\{{\frac{R - G}{C} + {4\ {if}\ M}} = B}\end{matrix}.} }}} & (29)\end{matrix}$

Equation (29) is designed so that a different degree is assigned foreach separate pure color as depicted in FIG. 25F. Also, the degeneratecolors like black, white and gray colors are assigned the degree 0°.

The computer system 100 can determine the luminance for each pixel as

${L = \frac{M + m}{2}}.$

Luminance can be used to estimate the power of the color or the amountof light contained in it. The computer system 100 can determine thesaturation as:

$\begin{matrix}{S = \{ {\begin{matrix}{{0{if}\ L} = 0} & {{{or}{\ }L} = 1} \\\frac{C}{1 - {❘{{2.L} - 1}❘}} & {otherwise}\end{matrix}.\begin{matrix}\  \\\ \end{matrix}} } & (30)\end{matrix}$

While Chroma represents an absolute color purity value, saturation canbe viewed as representing relative color purity. In someimplementations, the computer system 100 can determine or construct onlyhue color frames but not the saturation and luminance color frames.

Referring back to FIG. 25A, the computer system 100 can determine, foreach pixel position of an image of the sequence of images or of adownsampled image of the sequence of downsampled images, a correspondingpenalty score indicative of the similarity between a color intensityvalue of the pixel at the pixel position and the desired color value.The color value can represent a hue color value of the pixel of thepixel position and the desired color value can be a desired hue colorvalue. The desired hue color value can represent the hue color value offingers (or other body parts) when they are properly placed against thelens or opposite to (or facing) the photodetector 110.

The computer system 100 can compute or determine the penalty score foreach pixel of an image or a downsampled image as a function of theabsolute difference |H−T_(H)| between the hue value H of the pixel andthe desired hue color T_(H). Specifically, and considering the cyclicnature of the hue values as depicted in FIG. 25F, the computer system100 can compute or determine the penalty score for each pixel of animage or a downsampled image as a function of min(|H−T_(H)|,360−|H−T_(H)|). Multiple penalty scores can be defined based on pixelhue values.

FIG. 25G shows plots for three different penalty scores expressed asfunctions of pixel hue values, according to example embodiments. Thedesired hue value T_(H) is referred to in FIG. 25G as the target hue andis equal to 45. A first penalty score is referred to as the absolutepenalty and is defined as:

$\begin{matrix}{\Delta_{abs} = {\frac{1}{180}\min{( {{❘{H - T_{H}}❘},{360 - {❘{H - T_{H}}❘}}} ).}}} & (31)\end{matrix}$

Other penalty scores can be defined as functions of the absolutepenalty, e.g., Δ_(t)=f_(i)(Δ_(abs)). For instance, the saturatingpenalty Δ_(sat) (shown in FIG. 25G) can be viewed as a transformationfunction based on two parameters ρ₁ and ρ₂. The computer system 100 candetermine or compute the saturating penalty as:

$\begin{matrix}{\Delta_{sat} = \{ {\begin{matrix}{0,} & {{{if}\ \Delta_{abs}} < \rho_{1}} \\{\frac{\Delta_{abs}}{\rho_{2} - \rho_{1}},} & {{{if}\ \rho_{1}} < \Delta_{abs} < \rho_{2}} \\{1,} & {{{if}\ \rho_{2}} < \Delta_{abs}}\end{matrix}.} } & (32)\end{matrix}$

The smooth penalty Δ_(smooth) (also shown in FIG. 25G) can be defined asa transformation function of Δ_(abs) with two scale parameters μ and σ.The computer system 100 can determine or compute the smooth penaltyscore for each pixel of an image or a downsampled image as

$\begin{matrix}{\Delta_{smooth} = {{f_{smooth}( {{\Delta_{abs};\mu},\sigma} )} = {\tanh{( \frac{\Delta_{abs} - \mu}{\sigma} ).}}}} & (33)\end{matrix}$

In some implementations, the penalty score for each pixel of an image ora downsampled image can be defined in terms of a difference between thecolor value of the pixel and the desired color value, regardless ofwhether the pixel color value and the desired color value are hue valuesor other color values. For example, the desired color value can includedesired red, green and blue values that represent finger (or other bodypart) color. The computer system 100 can determine or compute thepenalty score for each pixel in terms of a distance (or differences)between the pixel red, green and blue values R_(pixel), G_(pixel) andB_(pixel) and the desired or target red, green and blue valuesR_(target), G_(target) and B_(target) (e.g., in terms of|R_(pixel)−R_(target)|, |G_(pixel)−G_(target)| and|B_(pixel)−B_(target)|). In some implementations, the computer system100 can use a different color space (e.g., other than the HSL colorspace) to determine or compute the penalty scores for different pixels.

The computer system 100 can arrange the penalty scores into a matrix Δhaving a size or dimension (d_(x), d_(y)). Note that the penalty matrixΔ described in this subsection has a different size or dimension thanthe penalty matrix described in subsection 8.2.

Referring back to FIG. 25A, the method 2500 can include the computersystem 100 determining, using penalty scores of the plurality of pixelpositions, a relative position of the body part of the user with respectto a desired position (2506). The computer system 100 can determine aposition vector (or orientation vector) indicative of the relativeposition of the finger or body part with respect to a desired positionin a similar way as discussed above in subsection D.2, e.g., with regardto equation (18). Specifically, the computer system 100 can determine orcompute the relative position (or position vector) of the finger or bodypart as the center of mass or barycenter of the penalty scores for thepixels of the image or the corresponding downsampled image, e.g., as

$\begin{matrix}{\delta = {\begin{pmatrix}\frac{\sum_{i,j}{i \cdot {\Delta( {i,j} )}}}{\sum_{i,j}{\Delta( {i,j} )}} \\\frac{\sum_{i,j}{j \cdot {\Delta( {i,j} )}}}{\sum_{i,j}{\Delta( {i,j} )}}\end{pmatrix}.}} & (34)\end{matrix}$

The computer system 100 can compute a position vector (or a relativeposition) of the finger or body part for each acquired image frame (orthe corresponding downsampled image frame). The computer system 100 mayfurther compute the magnitude and/or the angle (e.g., direction ororientation) of the position vector S as discussed above in subsectionD.2 with regard to equation (19).

The computer system 100 can provide a visual output indicative of therelative position for display on the display device 112 to assist orguide the users to correctly place their fingers (or other body part).For instance, the computer system can cause display of visual output asdiscussed above in subsection D.2, e.g., with regard to FIG. 18C. Thevisual output can include an indication of a classification of therelative position of the finger or body part. The computer system 100can determine the classification of the relative position of the fingeror body part as discussed above in subsection D.2, e.g., with regard toTable 3. The visual output can include an indication (e.g., a color bar)of a metric representing a quality of the placement of the finger orother body part on the lens of the camera.

The computer system 100 can cause display of the visual output for eachreceived image of the sequence of images. For instance, when the usermoves his or her finger (or body part) responsive to the visual output,the visual output determined based on following images will reflect themovement of the finger or body part.

D.9. Detection of Data Acquisition Conditions Using a Machine LearningModel

The approaches or processes described in subsections D.1 through D.8allow for detection of various conditions associated with theacquisition of optical (or transdermal) data of a user for use tomeasure the blood pressure of the user. The detected conditions canrelate to the positioning or placement of the finger (or other bodypart) of the user on the lens of the camera, physiological conditions ofthe user (e.g., cold finger, Arrhythmia or irregular heart beat) orlight conditions (e.g., dark or light saturation conditions), amongothers. These conditions (also referred to as error conditions) areindicative of potential issues or problems with the data acquisitionthat could lead to acquired data that wouldn't be good or reliable formeasuring or determining blood pressure or other vital signs. Thedetected conditions can be used by the computer system 100 to providevisual or other type of output for the user to help or instruct the usertake some action to enhance the quality of the acquired transdermaloptical data. It is to be noted that in all the approaches descried insection D the output provided to the use can be a visual output, audiooutput or some other type of output.

Each of the approaches in subsections D1-D8 can detect one or smallsubset of the possible conditions. To check for all or most of theconditions, one may need to use many of the approaches or methodsdiscussed in subsections D1-D8. In this subsection a more comprehensivemachine learning approach is described. The machine learning approachallows for detecting various conditions at once.

Referring now to FIG. 26 , a flowchart illustrating a method 2600 forenhancing acquisition of transdermal optical data is shown, according toexample embodiments. The method 2600 can include obtaining a sequence ofimages representing transdermal optical data of a subject and acquiredby a photodetector (STEP 2602). The method 2600 can include identifyinga plurality of image regions across the sequence of images (STEP 2604).The method 2600 can include generating a plurality of color intensitysignals associated with the plurality of image regions across thesequence of images (STEP 2606). The method 2600 can include determining,using a machine learning model and the plurality of color intensitysignals associated with the plurality of image regions, a conditionassociated with acquisition of the sequence of images (STEP 2608), andproviding feedback for presentation to a user based on the conditionassociated with the acquisition of the sequence of images (STEP 2610).

The method 2600 can include the computer system 100 or the application114 obtaining a sequence of images representing transdermal optical dataof a subject (STEP 2602), and identifying a plurality of image regionsacross the sequence of images (STEP 2604). The photodetector or camera110 can acquire the sequence of images as discussed above, for example,with regard to step 402 of FIG. 4 , step 1602 of FIG. 16 or step 1802 ofFIG. 18 . The acquisition of the sequence of image frames can beperformed, e.g., by the data acquisition module 502, in a similar way asSTEP 402 of method 400 in FIG. 4 . The computing device hosting thephotodetector 110 or another computing device can obtain the acquired orrecorded sequence of images. The computer system 100 can process imageframes of the sequence of images sequentially as they are received.

Identifying the plurality of image regions across the sequence of imagescan include the processing module 504 downsampling, for each acquiredimage frame, the respective color frames (e.g., R and G frames) toobtain corresponding downsampled color frames. In some implementations,the downsampled color frames can have a size of 9×9, 6×6 or other size.The computer system 100 can identify the plurality of image regionsacross the sequence of downsampled color frames. In someimplementations, the computer system 100 can forego downsampling andidentify the image regions within or across the original color frames ofthe sequence of images (e.g., not downsampled image frames). In someimplementations, the computer system can identify the plurality of imageregions across green and red color frames (or corresponding downsampledversions) of the acquired image sequence. In some implementations, thecomputer system 100 can identify the plurality of image regions acrosscolor frames (or corresponding downsampled versions) of other colorspaces (e.g., other than the RGB color space).

In some implementations, the color frames or the correspondingdownsampled versions in which the image regions are identified canbelong to a plurality of color spaces. For example, the computer system100 can identify the image regions across green, red and hue colorframes of the acquired image sequence. The computer system 100 candetermine or compute the hue color frames as discussed above with regardto equation (29). In general, the computer system 100 can applytransformations from one color space to one or more other color spacesto determine color frames across multiple color spaces.

Referring now to FIG. 27 , a diagram illustrating an example approach ofdividing a color (or image) frame (or a downsampled version thereof)2700 into a plurality of corresponding image regions. The pattern fordividing each color or image frame into the corresponding plurality ofimage regions can be the same (e.g., in terms of size, shape andlocation within the color frames) across various image or color frames.The image regions can include a central image region and one or moreside image regions. For instance, the signal-generating module 506 canidentify a central image region 2702, a left-side image region 2704, aright-side image region 2706, a top-side image region 2708, and abottom-side image region 2710 as depicted in FIG. 27 . The left-sideimage region 2704 can include one or more left columns of the image orcolor frame (or corresponding downsampled version) 2700, and theright-side image region 2706 can include one or more right columns ofthe image or color frame 2700. The top-side image region 2708 caninclude one or more top rows of the image or color frame (or thecorresponding downsampled version) 2700, and the bottom-side imageregion 2710 can include one or more bottom rows of the image region (orcorresponding downsampled version) 2700.

In some implementations, the computer system 100 can use a differentpattern for dividing the image or color frames into the correspondingimage regions. The number, the size and the shapes of the images regionsin each image frame can be different from the number, size and shapes ofthe image regions 2702, 2704, 2706, 2708 and 2710 of FIG. 27 . Forexample, the computer system 100 can identify a number of image regionsof equal size in each image or color frame.

The method can include determining, by the computing device, using amachine learning model and the plurality of color intensity signalsassociated with the plurality of image regions, a condition associatedwith acquisition of the sequence of images. The machine learning modelcan receive the plurality color intensity signals as input and providean indication of the condition as output. The method can includeproviding, by the computing device, feedback for presentation to a userbased on the condition associated with the acquisition of the sequenceof images.

Referring back to FIG. 26 , the method 2600 can include the computersystem 100 or the signal-generating module 506 generating a plurality ofcolor intensity signals associated with the plurality of image regionsacross the sequence of images (STEP 2606). The signal-generating module506 can generate each color intensity signal using a corresponding imageregion defined across color frames of a given color of the imagesequence. For instance, the computer system 100 can generate one or morered intensity signals using one or more corresponding image regionsdefied across red color frames (or corresponding downsampled versions)of the acquired image sequence, and/or one or more green intensitysignals using one or more corresponding image regions defied acrossgreen color frames (or corresponding downsampled versions) of theacquired image sequence. In some implementations, the computer system100 can generate one or more hue intensity signals using one or morecorresponding image regions defied across hue color frames (orcorresponding downsampled versions) of the acquired image sequence.

In some implementations, the computer system 100 or thesignal-generating module 506 can generate a color intensity signalassociated with an image region across the sequence of images byaveraging pixel color values of the image region for each image frame(or each corresponding color frame) of the sequence of images. In someimplementations, the computer system 100 or the signal-generating module506 may apply weighted averaging for an image region across color framesof the sequence of images to generate the corresponding color signal.

Referring back to FIG. 27 , the computer system 100 or thesignal-generating module 506 can average the pixel values in each of theimage regions 2702, 2704, 2706, 2708 and 2710, for both R and G colorframes (or corresponding downsampled versions) to generate correspondingred and green intensity signals. In some implementations, thesignal-generating module 506 can generate color signals for a subset ofthe image regions identified at step 2604 and/or a subset of the colorframes. For example, the signal-generating module 506 can generate oneor more green intensity signals using one or more identified imageregions (e.g., 2702, 2704, 2706, 2708 and 2710) across the green colorframes (or corresponding downsampled versions) of the sequence ofimages, one or more red intensity signals using one or more identifiedimage regions (e.g., 2702, 2704, 2706, 2708 and 2710) across the redcolor frames (or corresponding downsampled versions) of the sequence ofimages or a combination thereof. The signal-generating module 506 cangenerate one or more hue intensity signals using one or more identifiedimage regions (e.g., 2702, 2704, 2706, 2708 and 2710) across the huecolor frames (or corresponding downsampled versions) of the sequence ofimages. The color frame-image region pairs for which to generatecorresponding color intensity signals can be predefined.

The method 2600 can include the computer system 100 determining, using amachine learning model and the plurality of color intensity signalsassociated with the plurality of image regions, a condition associatedwith acquisition of the sequence of images (STEP 2608). The machinelearning model can receive the plurality of generated color intensitysignals as input and provide an indication of the condition as output.The machine learning model can be trained using a second plurality ofcolor intensity signals generated from one or more second imagesequences. The second plurality of color intensity signals can beassociated with the plurality of image regions across the one or moresecond image sequences or respective color frames. The color frames andthe images regions of the one or more second image sequences used togenerate the second plurality of color intensity signals (for trainingthe machine learning model) can correspond to or can be similar to theimage regions and the color frames used at steps 2604 and 2606. In someimplementations, the method 2600 can further include the computer system100 training the machine learning model using the second plurality ofcolor intensity signals.

The training data (e.g., the second plurality of color intensitysignals) can be labeled data. For instance various sets of trainingcolor intensity signals can be associated with known corresponding errorconditions (e.g., finger placement condition or state, blood perfusioncondition, light saturation or dark conditions, Arrhythmia, irregularheart rate, etc.). The machine learning model can include a neuralnetwork, a Bayesian network, a statistical model or other type ofmachine learning model that can be trained using training data. Themachine learning model, once trained, can receive a plurality of colorintensity signals corresponding to a plurality of predefined colorframe-image region pairs of a sequence of images, and generate one ormore indications of one or more conditions associated with acquisitionof the sequence of images.

The conditions can include conditions of the generated color signalsindicative of relative placement of the user finger (or other bodypart). For example, proper placement of the finger or body part canrequire that the central R signal (corresponding to central R imageregion 2702) to be greater (in corresponding values) than the central Gsignal (corresponding to the central image region 2702), and the centralR signal to be greater than the bottom-side G signal corresponding tobottom-side image region 2710. Proper placement of the finger or otherbody part may require the central G signal to be greater than theleft-side G signal (e.g., corresponding to left-side image region 2704).Proper placement of the finger or other body part can require thecentral G signal to be greater than the right-side G signal (e.g.,corresponding to the right-side image region 2706). Proper placement ofthe finger or other body part can require the central G signal to begreater than the top-side G signal (e.g., corresponding to the top-sideimage region 2708). Accordingly, the machine learning model ca detectimproper placement of the finger or other body part on the lens based onthe input color intensity signals generated at step 1606. The machinelearning model can detect other conditions, such as Arrhythmia,irregular heart rate, blood perfusion, or light saturation or darkconditions, among others, based on the plurality of color intensitysignals generated at step 2606 and provided as input to the machinelearning model.

Referring to FIGS. 28A-G, various color intensity signals 2802-2812illustrating various scenarios are shown, according to exampleembodiments. In each of FIGS. 28A-G, six color signals, mainly thecentral G signal 2802, the central R signal 2804, the top-side G signal2806, the bottom-side G signal 2808, the left-side G signal 2810 and theright-side G signal 2812 are shown. FIG. 28A illustrates a scenariowhere the user finger was removed from the lens during the dataacquisition process. The removed of the finger is reflected through asudden jump (or increase in intensity) in all the color intensitysignals 2802, 2804, 2806, 2808, 2810 and 2812.

FIGS. 28B and 28C show the color intensity signals 2802, 2804, 2806,2808, 2810 and 2812 for a low blood perfusion scenario and highperfusion scenario, respectively. The relative positions of these colorintensity signals are different in FIGS. 28B and 28C. For instance, inthe case of low blood perfusion (e.g., corresponding to cold finger),the central G signal 2802 is greater than the central R signal 2804 andthe magnitude of both signals is around 100. In the case of high bloodperfusion (FIG. 28C), the central G signal 2802 is smaller than thecentral R signal 2804 and the magnitude of both signals is highercompared to FIG. 28B.

FIG. 28D shows a scenario where the color intensity signals 2802, 2804,2806, 2808, 2810 and 2812 seem to be of good quality, but still lead toa failure of the OBPM model in providing a blood pressure measurement.

FIGS. 28E, 28F and 28G illustrate various lighting scenarios. FIG. 28Eshows the color intensity signals 2802, 2804, 2806, 2808, 2810 and 2812for a light saturation condition (e.g., high light intensity resultingin saturation in recorded color intensities). FIGS. 28F and 28G showsthe color intensity signals 2802, 2804, 2806, 2808, 2810 and 2812 fortwo relatively dark lighting conditions. The magnitudes and relativepositions of the color intensity signals 2802, 2804, 2806, 2808, 2810and 2812 in FIG. 28E are significantly different from those in FIGS. 28Fand 28G.

FIGS. 28A-G illustrate that the color intensity signals generated atstep 2606 can provide indications of various underlying conditions.Using training data, e.g., including a various sets of color intensitysignals and indications of corresponding underlying conditions, amachine learning model can be trained (e.g., by the computer system 100or other computer system) to detect various error condition (or dataacquisition conditions) based on color intensity signals generated asdescribed in step 2606. The trained machine learning model can receivethe plurality of color intensity signals (e.g., generated as discussedin relation with step 2606) and provide one or more indications of oneor more conditions.

The condition can include at least one of a condition related toplacement of a body part of the subject or user relative to thephotodetector or a blood perfusion condition (e.g., indicative of coldfinger or body part) of the subject. The condition can include alighting condition, such as a saturation or dark condition. In someimplementations, the condition can include an Arrhythmia conditionand/or an irregular heart rate condition.

Referring back to FIG. 26 , the method 2600 can include providing, bythe computing device, feedback for presentation to a user based on thecondition associated with the acquisition of the sequence of images(STEP 2610). The method 2600 can include the output module 512 or thecomputer system 100 providing feedback based on the determinedcondition(s). The output module 512 can provide feedback or instructionsfor presenting to the user via the application UI indicative of (orassociated with) the detected condition. The output module 512 canprovide general feedback or instructions to indicate that the finger isnot properly placed. The UI may provide the option (e.g., upon userselection) to render detailed instructions or a demo illustrating howthe user finger should be placed relative to the camera device 110and/or the flash device 206. In some implementations, the output module512 can present specific feedback or instructions depending on thedetected condition. For instance, if the finger is detected to beshifted to the left side of the camera 110, the output module 512 canpresent specific feedback to indicate the left shift of the finger, orcan instruct the user to move the finger slightly to the right.

The method 2600 or steps thereof can be performed by the computer system100 in combination with any of the methods described in this disclosure.In general, any combination of the methods or processes described inthis disclosure can be performed by the computer system 100.

E. Assessing PPG Signal Quality for Initiating Blood PressureMeasurement

Even when applying processes to enhance the accuracy or fidelity ofacquired optical transdermal data, e.g., as discussed with regard toSection C above, the corresponding generated PPG signal may still showdistortions and fail to accurately depict the pulse wave of the user. Toimprove the accuracy of measured blood pressure, the computer system 100or the signal quality assessment module 508 can check if the generatedPPG signal is suitable for biosensing analysis by verifying orinspecting one or more features of the PPG signal. In someimplementations, the computer system 100 or the signal qualityassessment module 508 can employ a threshold for a corresponding signalfeature to predict the quality of the PPG signal quality. The use ofthresholds allows for fast detection of poor signal quality, forexample, caused by low blood perfusion in the finger due to cold handsor generally low vascularity. The signal quality assessment module 508can determine based on the assessment of the quality of the PPG signalwhether or not to feed the PPG signal or portion thereof to the OBMmodule 510 for use to estimate blood pressure.

The PPG signal is expected to be periodic with a base frequency equal tothat of the user heart rate. However, the heart rate is not constantover time and evolves slowly due to respiratory rate or user activity,among other factors. The change in heart rate over time can induceartifacts in the frequency spectrum of the PPG signal, for example, withrespect to the locations of corresponding peaks. This fact, calls for asignal assessment approach that can estimate the local quality, e.g.,local period, of the PPG signal.

FIG. 29 shows a flowchart illustrating a method 2900 for assessing asignal quality for use to estimate blood pressure, according toinventive concepts of the current disclosure. In brief overview, themethod 2900 can include generating a color intensity signal using asequence of acquired transdermal images of a subject (STEP 2902), andcomputing a signal quality metric of the color intensity signal (STEP2904). The method 2900 can include determining an indication of aquality of the color intensity signal (STEP 2906), and providing theindication of the quality of the color intensity signal for display(STEP 2908).

In further detail, the method can include the computer system 100 or thesignal-generating module 508 generating the color intensity signal (STEP2902). The data acquisition module 502 can acquire the sequence oftransdermal images as discussed above with regard STEP 402 of FIG. 4 .The processing module 504 can identify an image block across thesequence of transdermal images, and the signal-generating module 506 cangenerate the color intensity signal based on the image block asdiscussed above with regard to STEPs 404 and 406 of FIG. 4 .

The method 2900 cam include the signal quality assessment module 508computing a signal quality metric of the color intensity signal (STEP2904). The signal quality metric of the color intensity signal caninclude a normalized autocorrelation metric or score as shown inequation (4) below, a peak or maximum of the normalized autocorrelationmetric or score within a predefined time window or time interval, aspectrum entropy based metric as described in equations (5) and (6)below or a combination thereof. The signal quality assessment module 508can compute a plurality of values of the signal quality metric. Forinstance, the signal quality assessment module 508 can evaluate thesignal quality metric iteratively using a different set of generatedsamples of the color intensity at each iteration.

In some implementations, the signal quality assessment module 508 cancompute a plurality of normalized autocorrelation metrics or parametersof the color intensity signal. Using the normalized autocorrelationmetrics, the computer system 100 or the signal quality assessment module508 can determine local features of the generated color intensity (orPPG) signal. For instance, the signal quality assessment module 508 cancompute the normalized autocorrelation parameters using a sliding windowof a given size n, where n is an integer representing the number ofsignal samples in the sliding window.

Referring to FIG. 30 , a diagram illustrating the use of a slidingwindow to compute the normalized autocorrelation metrics or parametersis shown, according to inventive concepts of the current disclosure. Thesignal 3002, referred to hereafter as x[.], represents the PPG signal.The windows 3004 a-3004 c represent different versions of the slidingwindow associated with different timestamps N−2, N−1 and N,respectively. In some implementations, the timestamp of each version ofthe sliding window can represent the index of the last signal sample inthat version of the sliding window. For instance, the last signal samplein the sliding window version 3004 c, having timestamp N and referred toas W_(N), is x[N], and the last signal sample in the sliding windowversion 3004 b, having timestamp N−1 and referred to as W_(N-1), isx[N−1].

For each time window W_(N), the computer system 100 or the signalquality assessment module 508 can compute a plurality of correspondingcovariance values Q_(N)(i) for a plurality of integer values i. For eachinteger i (0≤i≤n−1), the covariance value Q_(N)(i) can represent thecovariance between the window W_(N) and the window W_(N-1), and may bedescribed as:

$\begin{matrix}{{{Q_{N}(i)} = {\frac{1}{n}{\sum}_{j = 0}^{n - 1}( {{x\lbrack {N - j} \rbrack} - M_{N}} )( {{x\lbrack {N - i - j} \rbrack} - M_{N - i}} )}},} & (35)\end{matrix}$

where M_(N) represents the average of the signal values in the windowW_(N), and M_(N-i) represents the average of the signal values thewindow W_(N-i). Equation (29) can be re-written as:

$\begin{matrix}{{Q_{N}(i)} = {{\frac{1}{n}{\sum}_{j = 0}^{n - 1}{x\lbrack {N - j} \rbrack}{x\lbrack {N - i - j} \rbrack}} - {M_{N}{M_{N - i}.}}}} & (36)\end{matrix}$

In some implementations, the computer system 100 or the signal qualityassessment module 508 can compute the covariance values Q_(N)(i)iteratively to reduce the computational cost. Specifically, consideringequation (30), the signal quality assessment module 508 can compute thecovariance value Q_(N)(i) in terms of the covariance value Q_(N-1)(i) asin formula (31).

$\begin{matrix}{{Q_{N}(i)} = {{Q_{N - 1}(i)} + {\frac{1}{n}{x\lbrack N\rbrack}{x\lbrack {N - i} \rbrack}} + {{M_{N - 1}M_{N - 1 - i}} \pm {\frac{1}{n}{x\lbrack {N - n} \rbrack}{x\lbrack {N - n - i} \rbrack}}} - {M_{N}{M_{N - i}.}}}} & (37)\end{matrix}$

The iterative approach described in equation (31) reduces thecomputational complexity significantly. According to equation (31), thesignal quality assessment module 508 can compute each new covariancevalue Q_(N)(i) using only four multiplications and four additionsprovided that the values covariance values Q_(N-1)(i) and the meanvalues M_(N-1) through M_(N-n) are stored in the memory 104. As such,the gain in computational complexity, when using equation (31) insteadof equation (30), is in the order of n; the size of the sliding windows.In fact, the computational cost of computing the covariance values Q_(N)(i) for all N and i is of order O(N. K) operations, where Nis the sizeof the signal and K the number of delayed windows for which we want tocompute autocorrelation.

The signal quality assessment module 508 can then compute, for each timewindow W_(N), the corresponding normalized autocorrelation metrics orparameters, as:

$\begin{matrix}{{{q_{N}(i)} = \frac{Q_{N}(i)}{\sqrt{Q_{N}(0)}\sqrt{Q_{N - i}(0)}}},} & (38)\end{matrix}$

for each integer i where 0≤i≤n−1.

Referring now briefly to FIGS. 31A and 31B, FIGS. 31A and 31B show twoexamples of filtered PPG signals and the corresponding normalizedautocorrelation metrics or parameters q_(N)(i) for a given time windowW_(N). The PPG plot at the left side of FIG. 31A represents an exampleof a good quality PPG signal, while the plot at the right side of FIG.31A represents an example of a bad PPG signal. In FIG. 31B, the plot onthe left represents the normalized autocorrelation metrics or parametersq_(N)(i) for the left PPG signal of FIG. 31A. The plot on the right ofFIG. 31B represents the normalized autocorrelation metrics or parametersq_(N)(i) for the right PPG signal of FIG. 31A. If the PPG signal is ofgood quality, the corresponding q_(N)(i) has several successive valleysand peaks reaching very high correlation values close to 1.

In some implementations, the computer system 100 or the signal qualityassessment module 508 can define values around the peak, e.g., as apolynomial of degree 2, and interpolate the values of q_(N)(i) aroundthe peak with such a polynomial, to achieve sub-frame precision. Thecomputer system 100 or the signal quality assessment module 508 can thenextract a sub-frame maximum by finding the maximum of the interpolationfunction. In FIG. 31B, the point 3102 represents the first peak ofq_(N)(i) for the good-quality PPG signal. In FIG. 31B (b), no peak ofq_(N)(i) for the bad-quality PPG signal, other than the first peak,reaches a high value close to 1.

In some embodiments, the computer system 100, or the signal qualityassessment module 508, can employ a signal quality metric based onspectrum entropy. A periodic signal can be viewed as composition of unitpure periodic signals or sinusoids that can be considered as symbols.Based on information theory, the amount of information contained in thissuperposition of symbols can be related to Shannon entropy.Specifically, the signal quality metric based on spectrum entropy can bedescribed as:

H(X)=Σ_(k=0) ^(N-1) {circumflex over (P)}[k]log {circumflex over(P)}[k],  (39)

where {circumflex over (P)}[k] represents the normalized power spectrumof the Fourier transform {circumflex over (X)}[k] at the frequency f_(k)and can be defined as:

$\begin{matrix}{{\overset{\hat{}}{P}\lbrack k\rbrack} = {\frac{{❘{\overset{\hat{}}{X}\lbrack k\rbrack}❘}^{2}}{{\sum}_{l = 0}^{N - 1}{❘{\overset{\hat{}}{X}\lbrack l\rbrack}❘}^{2}}.}} & (40)\end{matrix}$

The computer system 100 or the signal quality assessment module 508 cancompute H(X) according to equations (33) and (34).

In some implementations, the computer system 100 or the signal qualityassessment module 508 can compute a plurality of values of H(X)iteratively as the camera 110 acquires new image frames. For example,when computing the Fourier transform X, the computer system 100 or thesignal quality assessment module 508 can use a different set, or adifferent sliding window, of samples of the color intensity signal ateach iteration. The computer system 100 or the signal quality assessmentmodule 508 can compute, for each computed X, a corresponding value ofH(X). As such, the computer system 100 or the signal quality assessmentmodule 508 does not need to wait for acquisition of the whole colorintensity signal to compute H(X), but can evaluate H(X) iteratively asnew samples of the color intensity signal (or PPG signal) are generated.

The method 2900 can include the computer system 100 or the signalquality assessment module 508 determining the indication of the qualityof the color intensity signal (STEP 2906). The indication of the qualityof the color intensity signal can be a signal quality level, such aspoor or good. The computer system 100 or the signal quality assessmentmodule 508 can determine the indication of the quality of the colorintensity signal using the signal quality metric. For instance, thecomputer system 100 or the signal quality assessment module 508 candetermine the quality level of the color intensity signal using a peakof the normalized autocorrelation score or parameter q_(N)(i) over eachtime window W_(N). Peaks of q_(N)(i) for each time window W_(N) usuallyoccur at the local period of heart pulsation of the user or subject.

The autocorrelation array q_(N)(i) always has a value of 1 at its firstentry because a signal window (e.g., window 3004 c) has a correlation of1 with itself. When determining the quality indication, is to find thenext peak in the autocorrelation array q_(N)(i). One approach is todetect the second in height (e.g., the maximum of the array q_(N)(i)without the first entry) and take it as the autocorrelation peak. Thecomputer system 100 or the signal quality assessment module 508 candetermine or identify the second highest peak of the autocorrelationarray q_(N)(i) for each time window. FIG. 31C shows an exampleautocorrelation array where the second highest peak is used to detectthe local period of the heart pulsation. However, such approach (e.g.,second highest peak) may not always accurately detect the local periodof the heart pulsation. Referring to FIG. 31D, plots of two exampleautocorrelation arrays are shown. For the autocorrelation array on theleft, the second highest peak occurs at the local period of the heartpulsation. However, for the autocorrelation array on the right, thesecond highest peak does not coincide with the local period of the heartpulsation because the color intensity signal contains harmonics of highorder. In fact, FIG. 31D shows a degenerate case where the secondhighest peak is not the peak of interest, and illustrates the importanceof a penalization of the peaks with higher index.

To avoid errors in detecting the peak of interest (e.g., due to highersignal harmonics), the computer system 100 or the signal qualityassessment module 508 can determine or identify the index of the peak ofinterest in the autocorrelation array q_(N)(i) as:

$\begin{matrix}{{\hat{l} = {\underset{i}{\arg\max}\frac{q(i)}{i^{\theta}}}},} & (41)\end{matrix}$

where θ>0, and i^(θ) is an equalizing factor that penalizes peaks thatare too far from the first peak (or first entry) of the autocorrelationarray q_(N)(i). The penalization avoids errors in estimating the periodof the pulses.

The computer system 100 or the signal quality assessment module 508 cancompare the peak of q_(N)(i) within each time window W_(N) to acorresponding threshold value, such as 0.85, 0.9 or other value smallerthan but close to 1. The computer system 100 or the signal qualityassessment module 508 can determine that at least a portion of the colorintensity signal to be of poor quality if the peak of q_(N)(i) issmaller than the threshold value for one or more consecutive timewindows. For example, the computer system 100 or the signal qualityassessment module 508 can determine that at least a portion of the colorintensity signal to be of poor quality if the peak of q_(N)(i) issmaller than the threshold value for a sequence of consecutive timewindows corresponding to a predefined data acquisition period or numberof consecutive seconds. In some implementations, the computer system 100or the signal quality assessment module 508 can determine that at leasta portion of the color intensity signal to be of poor quality if thepeak of q_(N)(i) is smaller than the threshold value for one or moreconsecutive time windows, or if the variation in the local pulsationperiod exceeds a predefined variation threshold.

In some implementations, the computer system 100 or the signal qualityassessment module 508 can compare each computed value of H(X) to anentropy threshold, such as −0.06, −0.041 or another number smaller thanbut close to zero. The computer system 100 or the signal qualityassessment module 508 can determine that at least a portion of the colorintensity signal to be of poor quality if one or more values (or amultiple consecutive values) of H(X) are determined to be smaller thanthe entropy threshold. In some embodiments, the computer system 100 orthe signal quality assessment module 508 can determine that at least aportion of the color intensity signal to be of poor or good qualitybased on the comparison of a peak of q_(N)(i) within a time window W_(N)to the corresponding threshold value, variation in the local pulsationperiod, comparison of a value of H(X) to the entropy threshold or acombination thereof.

FIG. 32 shows an image depicting individual normalized autocorrelationparameters q_(N)(i) for a plurality of time windows W_(N). Thehorizontal axis represents the timestamps N of the time windows. Thevertical axis represents the delay i between a current window W_(N) anda comparative window W_(N-i). The brighter a pixel with coordinates (N,i), the more correlated are the windows W_(N) and W_(N-i). The line 1402represent the peak of q_(N)(i) for each time window W_(N).

The method 2900 can include the computer system 100 or the signalquality assessment module 508 providing the indication of the quality ofthe color intensity (or PPG) signal for display. The computer system 100can continuously display indications of the signal quality as the camera110 acquires new video frames. The display of the indications of thesignal quality during the data acquisition process allows for detectionof inadequate placement of the finger or other body part over the cameralens, and providing immediate notifications to the user of the computersystem 100. As such, the user can quickly adjust the placement ofhis/her finger (or other body part) over the camera lens.

In some embodiments, the computer system 100 or the signal qualityassessment module 508 can abort the data acquisition process (e.g.,acquisition of the image frames and generation of samples of the colorintensity signal) if the signal quality is determined to be poor (or oflow level) for a predefined cumulative time period. For example, thecomputer application 114 can be configured to cause the computer system100 to acquire data for a total of 30 seconds. To generate a reliableand accurate measurement of blood pressure or pulse, at least 60% of themeasured color intensity (or PPG) signal should be of good quality. Assuch, if at any point during the data acquisition period the generatedcolor intensity (or PPG) signal is determined to be of poor (or low)quality over a cumulative time period of 12 seconds or more (e.g., 40%of 30 seconds), the computer system 100 or the signal quality assessmentmodule 508 can abort the data acquisition (or blood pressure measuring)process, and display a message or indication to the user about thetermination of the process.

Referring to FIG. 34 , a flowchart illustrating a data acquisitionmethod 3400 is shown, according to example embodiments. When dataacquisition is initiated or triggered, the computer system 100 can startor set a timer (STEP 3402). The computer system 100 can iterativelyrecord or acquire image frames (STEP 3404). For each recorded imageframe, the computer system can compute and store a corresponding qualityvalue or quality indication (STEP 3406) and compare the quality value toa quality threshold (STEP 3408). If the quality value is below thethreshold, the computer system can increment a counter of poor qualityframes and check whether the counter exceeds a maximum allowed number ofpoor quality frames (STEP 3410). If the counter exceeds the maximumallowed number of poor quality frames, the computer system 100 can abortthe data acquisition and display an error message indicating poorquality data (STEP 3412). If the quality is found to exceed the qualitythreshold at STEP 3408 or if the counter does not exceed the maximumallowed number of poor quality frames at STEP 3410, the computer system100 can proceed to STEP 3414 and check if the timer reached a predefinedtime duration (e.g., 30 second). If yes the computer system can stop thetime and end data acquisition (as a successful acquisition), otherwisethe computer system 100 loop back to STEP 3404 to acquire a new imageframe.

Referring to FIGS. 35A-L, screenshots of a user interface (UI) of thecomputer application 114 at various phases of the acquisition oftransdermal optical data are shown, according to example embodiments ofthe current disclosure. The application 114, when executed, can causethe computer system 100 to display instructions and/or feedback to theuser during in association with the data acquisition process. FIG. 35Ashows a screenshot of an example home screen. When the computerapplication 114 is started or initiated on the computer system 100,e.g., by user action, the computer application 114 can cause thecomputer system 100 to display the UI shown in FIG. 35A. The UI caninclude the last blood pressure measurement and the data and time ofsuch measurement. The UI may show a plot or graph of previousmeasurements of the user. The UI may include a tab or icon, e.g., the“Measure” tab, to initiate a new measuring process of blood pressure.For instance, the user can click or tap on the “measure” icon to triggera new measuring process.

Upon actuation of the “Measure” tab or icon, the computer application114 can cause the computing device to display the instructions UI inFIG. 35B. The instructions UI can include instructions or directions forthe user regarding how place a finger or other body part against thecamera 110 of the computer system 100. The instructions or directionscan include an image depicting an example or a desired placement of thefinger or other body part against the camera 110. The instructions orguidelines can include textual instructions explaining how to place thefinger or other body part against the camera 100. The instructions UIcan present an option to play or watch, e.g., via the “Watch tutorialvideo” tab or icon, a tutorial video that depicts finger (or other bodypart) placement and/or other instructions/information on how to use thecomputer application 114. The instructions UI can include a selectableicon to skip the instructions UI in future measurements of the bloodpressure. The instructions UI can include an interactive item, e.g., the“Next” tab or icon, to initiate the measuring process when the user isready.

Upon initiation of the measuring process, the computer application 114can cause the computing device 110 to detect PPG signal quality, e.g.,using methods or techniques discussed above, and display arepresentation of a quality meter depicting an indication or value ofthe quality of measured PPG signal. The screenshots in FIGS. 35C-I showinstances of feedback UI at various time instances of the measuringprocess. The feedback UI can include the visual representation of thequality meter, e.g., the disc including a signal quality bar. In someimplementations, the PPG signal quality as depicted by the signalquality bar can vary between poor and good. For example, the computersystem 100 may consider the PPG signal quality to be good when thenormalized autocorrelation score q_(N)(i) is over 85% (or over 0.85). Insome implementations, the signal quality bar can depict poor signalquality with blue color and/or the term “Poor” and depict good signalquality with green color and/or the term “Good.”

In some implementations, the computer system 100 can start recording themeasured PPG signal once the quality first reaches the “Good” level,e.g., when the normalized autocorrelation score q_(N)(i) first exceeds85%. Referring to FIGS. 35C and 35D, screenshots of the UI depicting PPGsignal detection leading to potential start of signal recording.Initially, the feedback UI or the quality meter can indicate that thecomputer system 100 is in the process of “Detecting pulse” whiledetected PPG signal is still poor.

Once the measured PPG signal quality reaches a good quality level, thefeedback UI can indicate the start of PPG signal recording, for example,by rendering the term “Measuring” as part of the quality meter, asillustrated in FIGS. 35E and 35F. The quality meter can display, via thesignal quality bar, the current detected signal quality. The feedback UImay also display a plot or graph of a most recent recorded portion ofthe PPG signal. Throughout the signal detection and recording process,the feedback UI can render a text (or other visual indication) to remindthe user to keep the finger (or other body part) over the camera lenswith gentle and even pressure. After the signal recording processstarts, the quality meter can display a timer, e.g., a decreasingcounter, indicating time left in the data acquisition process.

If the detected signal quality goes below 85% for a predefined timeduration, e.g., x consecutive seconds, the quality meter or the signalquality bar can display an indication of the poor quality. As such, theuser is notified of the poor quality and can promptly adjust the finger(or other body part) placement relative to the camera lens. The feedbackUI can continue to render the plot or graph of the recorded PPG signaleven if the detected quality is poor, as illustrated in FIGS. 35G-I.

If the signal quality is poor for a predefined cumulative time period,e.g., y cumulative seconds, the computer system 100 can stop the signalrecording process. Once the signal recording is stopped, the computerapplication 114 can cause the computer system 100 to display or render amessage indicative of the termination of the signal recording. FIG. 35Jshows an example of such message displayed by the computer system 100.The message can include an interactive item, e.g., icon, allowing theuser to start or initiate a new signal recording process. If the userinteracts with the interactive item, the computer system 100 can restartthe signal detection and signal recording process, and displaycorresponding UIs as illustrated in FIGS. 35K and 35L. If the measuredPPG signal has a good quality for at least a predefined cumulative timeduration, e.g., 60% or more of the total recording duration, thecomputing device can use the recorded signal to measure the user's bloodpressure and/or the user's pulse and display the measurements in ameasurement UI. For example, the total recording duration can be equalto 30 seconds, and the computer system 100 can keep track of the signalquality metric(s) during this time period. Considering the predefinedcumulative time duration, the computer system 100 may abort signalrecording if bad signal is detected for a cumulative time period of 12seconds.

In some implementations, the computer system 100 can record or storedisplayed feedback, plots of measured signal, computed normalizedautocorrelation score q_(N)(i) and/or other parameters in a document ordata structure, such as a Firebase database or data storage. Thedocument can include a tab “Poor Signal Quality” for accessinginformation related to poor signal quality, and/or a tab “End ofmeasurements” to access information related to termination of PPG signalrecording, such as time of termination or reason for termination, amongothers. The document or data structure can be maintained in the computersystem 100 or on a remote computer server.

In some implementations, the computer application 114 can cause thecomputer system 100 to notify the user, e.g., via the feedback UI, if adetection is made indicating that that the finger (or other body part)is not placed correctly or is not fully covering the camera lens. The UImay instruct the user to adjust the placement of his/her finger or bodypart, or indicate to which direction the finger or body part is to beshifted. The computer application 114 may be configured to detect fingertremor that prevents recording of a good quality PPG signal, and notifythe user of such fact via the feedback UI. In some implementations, thecomputer application 114 may be configured to detect that the signalamplitude is weaker than normal, e.g., finger feels cold and blood flowin the finger is sub-optimal, and notify the user the user via thefeedback UI. In general, the computer application 114 may be configuredto provide detailed feedback to the user that specifies the reason orcause of a bad quality signal.

F. Improving Accuracy of Blood Pressure Measurements Using Smart OffsetCalibration Based on Reference Signals Generated Using a Blood PressureMeasurement Device

The OBPM algorithm used to estimate blood pressure measurements based onthe recorded (or generated) PPG signal, is able to accurately trackblood pressure change, but does not provide accurate measurements (orestimates) of the absolute pressure. This calls for the use ofcalibration processes to enhance the accuracy of measurements ofabsolute blood pressure.

A classical calibration approach includes determining a calibrationoffset, and adding the determined calibration offset to each bloodpressure measurement. FIG. 36 shows a block diagram of a classicalcalibration model 3600. The computing system can obtain a calibrationblood pressure measurement, referred to herein as reference calibration3602 or Cuff_(calibration), that is provided by a cuff-based bloodpressure measuring device. The computer system 100 can generate,approximately at the same time or within a short time period ofmeasuring the calibration reference 3602, a second calibration bloodpressure measurement 3604 generated using a calibration PPG signal(e.g., determined using the OBPM algorithm). The second calibrationsignal 3604 is referred to herein as OBPM_(calibration). The classicalcalibration model 3600 can receive Cuff_(calibration) andOBPM_(calibration) as input calibration parameters, and determine acalibration offset as the difference between Cuff_(calibration) andOBPM_(calibration). When the classical calibration model 3600 receives anew blood pressure measurement 3606 (denoted as OBPM_(i)), e.g.,estimated by the OBPM algorithm using a recording PPG signal, thecomputer system 100 or the calibration model 3600 can compute acalibrated blood pressure measurement 3608 (denoted as BP_(i)) as:

BP _(i)=OBPM_(i)+(Cuff_(calibration)−OBPM_(calibration)).  (42)

Equation (42) can be viewed as a simple linear regression with a weightof +1 for OBPM_(i) and Cuff_(calibration) and a weight for −1 forOBPM_(calibration). In some implementations, multiple calibrationoffsets determined based on multiple calibration pairs(Cuff_(calibration), OBPM_(calibration)) can be used.

In the current disclosure, more complex calibration models are used toimprove the accuracy of absolute blood pressure measurements based onrecorded PPG signals. Referring to FIG. 37 , a block diagramillustrating a calibration system 3700 is shown, according to exampleembodiments. The calibration system 3700 can include an OBPM module (oralgorithm) 3702 and calibration module 516. The calibration module 516can include a feature extraction module 3704 and an absolute bloodpressure estimation module 3706 (e.g., calibration model, calibrationfree estimation model or classification model). The calibration module516 can receive calibration measurements, e.g., Cuff_(calibration) andOBPM_(calibration), a calibration PPG signal, user demographicinformation, a recording PPG signal and a blood pressure estimate/valuegenerated by the OBPM module 3702 based on recording PPG signal asinput, and provide the systolic and/or diastolic blood pressuremeasurement(s) as output.

Referring to FIG. 38 , a flowchart of a calibration method 3800 isshown, according to example embodiments. In brief overview, the method3800 can include associating a calibration PPG with one or moremeasurement values generated by a blood pressure measurement device(STEP 3802), and obtaining a recording PPG signal (STEP 3804). Themethod 3800 can include identifying a blood pressure calibration modelbased on the calibration PPG signal and the recording PPG signal (STEP3806), and generating a calibrated blood pressure value using therecording PPG signal, features associated with the calibration PPGsignal and the identified blood pressure calibration model (STEP 3808).

FIG. 39 shows a block diagram illustrating a blood pressure calibrationmodel 3900, according to example embodiments. The calibration model 3900is also referred to herein as a smart offset calibration model 3900. Thecalibration model 3900 can be a linear regression model, a nonlinearregression model, a RandomForest regressor, a support vector machineregressor, a multi-layer perceptron or a neural network among others.

Referring now to FIGS. 37-39 , the method 3800 can include the computersystem 100 or the calibration module 516 associating a calibration PPGsignal generated from a first sequence of image frames obtained fromphotodetector 110 of the computer system 100 with one or morecalibration measurement values (or calibration references) 3902generated by a blood pressure measurement device different from thecomputer system 100 (STEP 3802). The one or more calibration measurementvalues or calibration references 3902 (e.g., systolic and/or diastolic)can represent blood pressure measurement(s) of a user of the computersystem 100 generated by a cuff based blood pressure measurement device.The calibration reference(s) can be referred to as Cuff_(calibration).The computer system 100 can generate a calibration PPG signal of theuser at the same time or within a short time period of measuring thecalibration reference(s) 3902 or Cuff_(calibration). The calibration PPGsignal can be generated by the computer system as discussed above withregard to FIGS. 4 and 7 . The computer system 100 or the calibrationmodule 516 can associate the calibration reference(s) 3902 orCuff_(calibration) to the calibration PPG signal. For instance, thecomputer system 100 can store the calibration PPG signal in associationwith the calibration reference(s) 3902 or Cuff_(calibration), e.g.,using a data structure. The association can indicate that both thecalibration PPG signal and the calibration reference(s) 3902 orCuff_(calibration) were generated simultaneously or within a short timeperiod (e.g., with 5 minutes or within 10 minutes) of each other. Thecalibration module 516 receive both the calibration PPG signal and thecalibration reference(s) 3902 or Cuff_(calibration) as input calibrationdata.

The method 3800 can include the computer system 100 obtaining arecording PPG signal (or new PPG in FIG. 39 ) generated from a secondsequence of image frames obtained from the photodetector 110 (STEP3804). The second sequence of image frames can be acquired at a latertime by the computer system 100 for measuring the blood pressure of theuser. The computer system 100 can generate the recording PPG signalusing the second sequence of image frames as discussed above with regardto FIGS. 4 and 7 . The calibration module 516 can receive the recordingPPG signal as input recording data for measuring the blood pressure ofthe user.

The method 3800 can include the computer system 100 or the calibrationmodule 516 identifying a calibration model from a plurality of bloodpressure calibration models based on the calibration PPG signal and therecording PPG signal (STEP 3806). The computer system 100 or thecalibration module 516 can maintain or have access to the plurality ofblood pressure calibration models. The plurality of blood pressurecalibration models can include one or more linear regression models, oneor more nonlinear regression models, one or more trained machinelearning models or a combination thereof. The plurality of bloodpressure calibration models can include one or more one or more trainedmachine learning regression models.

The computer system 100 or the calibration module 516 can identify orselect the calibration model, from the plurality of blood pressurecalibration models, based on outputs of the OBPM module 3702 when fedwith calibration PPG signal or the recording PPG signal as input. TheOBPM module 3702 can receive a PPG signal (raw PPG signal) or alogarithmic PPG signal as input. A logarithmic PPG signal is a PPGsignal to which a logarithmic function or transformation is applied. Thecomputer system 100 or the OBPM module 3702 can generate, from therecording PPG signal, a logarithmic recording PPG signal, and generate,from the calibration PPG signal, a logarithmic calibration PPG signal.The OBPM module 3702 can separately use each of the calibration PPGsignal, the logarithmic calibration PPG signal, the recording PPG signaland the logarithmic recording PPG signal as input and estimate (orattempt to estimate) a corresponding blood pressure measurement for eachof these input signals. However, the OBPM module 3702 may not besuccessful in estimating a corresponding blood pressure measurement foreach of these input signals.

The computer system 100 or the calibration module 516 can determine (i)whether the recording PPG signal generates a first blood pressureestimate, (ii) whether the logarithmic recording PPG signal generates asecond blood pressure estimate, (iii) whether the calibration PPG signalgenerates a third blood pressure estimate, and (iv) whether thelogarithmic calibration PPG signal generates a fourth blood pressureestimate. The computer system 100 or the calibration module 516 canidentify or select the calibration model from the plurality of bloodpressure calibration models based on the events (i), (ii), (iii) and(iv). For instance, the computer system 100 or the calibration module516 can identify or select the calibration model from 12 different bloodpressure calibration models according to Table 4 below.

TABLE 4 Events for Calibration Model Selection/Identification. Recordingsignal Group_raw& log Group_raw Group_log Group_noObpm CalibrationGroup_raw&log_calib Model 1 Model 2 Model 3 Model 4 signalGroup_raw_calib Model 5 Model 6 Model 7 Model 8 Group_log_calib Model 9Model 10 Model 11 Model 12

The term “Group_raw&log_calib” means that the OBM module 3702 providesblood pressure estimates for both the raw calibration PPG signal and thelogarithmic calibration PPG signal. The term “Group_raw_calib” meansthat the OBM module 3702 provides a blood pressure estimate only for theraw calibration PPG signal but not for the logarithmic calibration PPGsignal. The term “Group_log_calib” means that the OBM module 3702provides a blood pressure estimate only for the logarithmic calibrationPPG signal but not for the raw calibration PPG signal. The term“Group_raw&log” means that the OBM module 3702 provides blood pressureestimates for both the raw recording PPG signal and the logarithmicrecording PPG signal. The term “Group_raw” means that the OBM module3702 provides a blood pressure estimate only for the raw recording PPGsignal but not for the logarithmic recording PPG signal. The term“Group_log_calib” means that the OBM module 3702 provides a bloodpressure estimate only for the logarithmic recording PPG signal but notfor the raw recording PPG signal. The term “Group_noOBPM” means that theOBM module 3702 fails to provide an estimate either for the recordingPPG signal or for the logarithmic recording PPG signal.

The method 3800 can include the computer system 100 or the calibrationmodule 516 generating a calibrated blood pressure value 3918 using therecording PPG signal, features associated with the calibration PPGsignal and the identified calibration model (STEP 3808). Each bloodpressure calibration model of the plurality of blood pressurecalibration models can be associated with a corresponding set ofparameter variables used to determine calibrated blood pressure values.The corresponding set of parameter variables for each blood pressurecalibration model M_(j) can be indicative (or can represent) a set offeatures that are provided as input to the blood pressure calibrationmodel M and are used by the calibration model M to determine the bloodpressure value(s) (e.g., systolic and/or diastolic) of the user. Forexample, a blood pressure calibration model M_(j) can be a linearregression model where the blood pressure value of the user isdetermined as:

BP _(i) =M _(j)(feature₁, . . . ,feature_(n) _(j) )=β₀ ^(j)+Σ_(k=1) ^(n)^(j) β_(k) ^(j)feature_(k).  (43)

The parameter variables feature₁, . . . , feature_(n) _(j) represent theset of features used by the blood pressure calibration model M_(j) toestimate or determine the absolute calibrated blood pressure values,such as the absolute calibrated blood pressure value BP_(i) of recordingi. The parameters β₀ ^(j), . . . , β_(n) _(j) ^(j) represent the linearregression coefficients of the linear regression model M_(j), and n_(j)represents the total number of features associated with the calibrationmodel M_(j). In some implementations, the blood calibration model can bea linear regression model, a nonlinear regression model, a neuralnetwork, a machine learning model or a combination thereof.

The set of features of a blood pressure calibration model, such as modelM_(j), can include systolic and/or diastolic blood pressure estimatesgenerated by the OBPM module 3702 responsive to an input signal equal tothe raw recording PPG signal or the logarithmic recording PPG signal.For instance, the set of features can include a systolic blood pressureestimate generated using the recording PPG signal as an input signal, adiastolic blood pressure estimate generated using the recording PPGsignal as an input signal, a systolic blood pressure estimate generatedusing a logarithmic recording PPG signal as input signal, a diastolicblood pressure estimate generated using the logarithmic recording PPGsignal as input signal or a combination thereof.

The set of features of a blood pressure calibration model, such as modelM_(j), can include calibration features (or OBPM calibrationmeasurements) such as systolic and/or diastolic blood pressure estimatesgenerated by the OBPM module 3702 responsive to an input signal equal tothe raw calibration PPG signal or the logarithmic calibration PPGsignal. For instance, the set of features can include a systolic bloodpressure estimate generated using the calibration PPG signal as an inputsignal, a diastolic blood pressure estimate generated using thecalibration PPG signal as an input signal, a systolic blood pressureestimate generated using a logarithmic calibration PPG signal as inputsignal, a diastolic blood pressure estimate generated using thelogarithmic calibration PPG signal as input signal or a combinationthereof. The calibration features can include a systolic calibrationreference and/or a diastolic calibration reference representingcalibration blood pressure measurements provided by the cuff baseddevice. The calibration features are listed in Table 5 below.

TABLE 5 Calibration Features. Name Definition Raw Sys_calib Systolicoutput of OBPM for the calibration with the input being the rawcalibration PPG signal Raw Dia_calib Diastolic output of OBPM for thecalibration with the input being the raw calibration PPG signal LogSys_calib Systolic output of OBPM for the calibration with the inputbeing the log of the calibration PPG signal Log Dia_calib Diastolicoutput of OBPM for the calibration with the input being the log of thecalibration PPG signal Ref Sys_calib Systolic references from thecalibration Ref Dia_calib Diastolic references from the calibration

The set of features of a blood pressure calibration model, such as modelM_(j), can include global PPG signal features of the recording PPGsignal and/or global PPG signal features of the calibration PPG signal.The feature extraction module 3704 can extract global PPG signalfeatures of the recording PPG signal from the logarithmic recording PPGsignal, and can extract global PPG signal features of the calibrationPPG signal from the logarithmic calibration PPG signal. The featureextraction module 3704 can determine the absorbance by computing thelogarithm of a PPG signal, and normalize the absorbance by subtractingthe corresponding mean across time and divide by the standard deviationacross time to remove the effect of hardware configuration, distancefrom light source, intensity of light source or environment, amongothers. Considering Beer's Law

A(θ,t)=A _(tissue)(θ)+A _(veinous)(θ)+A _(arterial)(θ,t)+A_(others)(θ)=A _(stat.)(θ)+a _(art.blood)(θ)L(t)  (44)

where A_(tissue)(θ), A_(veinous)(θ), A_(others)(θ) are the absorbance ofstationary elements of the finger that can be aggregated in A_(stat).(0), A_(arterial)(0, t) is the time dependent contribution of thearterial blood flow, a_(art.blood) is the absorptivity constant ofarterial blood, and L(t) is the light path length depending blood flowand thus on time. With regard to the setting for measuring bloodpressure the light path length depend on the relative position of thelight source 108 and the photodetector 110, which makes it devicespecific and does not vary over time during a recording. In the otherhand the concentration of the solution (or blood) depends on the amountof blood in the vessel, which varies over time.

The feature extraction module 3702 can apply the normalization to theBeer Law using the following equations:

$\begin{matrix}{{A(t)} = {A_{s{{tat}.}} + {{a_{{art}.{blood}}(\theta)}{{L(t)}.}}}} & (45)\end{matrix}$ $\begin{matrix}{\overset{\_}{A} = {{\frac{1}{n}{\sum}_{t = 1}^{n}{A(t)}} = {{\frac{1}{n}{\sum}_{t = 0}^{n}( {A_{s{{tat}.}} + {{a_{{art}.{blood}}(\theta)}{L(t)}}} )} = {A_{s{{tat}.}} + {{a_{{art}.{blood}}(\theta)}{\overset{¯}{L}.}}}}}} & (46)\end{matrix}$ $\begin{matrix}{{\sigma(A)} = {\sqrt{\frac{1}{n}{\sum}_{t = 1}^{n}( {{A(t)} - \overset{¯}{A}} )^{2}} = {{a_{{art}.{blood}}(\theta)}{{\sigma(L)}.}}}} & (47)\end{matrix}$ $\begin{matrix}{\frac{{A(t)} - \overset{¯}{A}}{\sigma(A)} = {\frac{{L(t)} - \overset{¯}{L}}{\sigma(L)} = {\frac{A_{{stat}.} + {{a_{{art}{blood}}(\theta)}{L(t)}} - A_{stat} - {{a_{{art}{blood}}(\theta)}\overset{\_}{L}}}{{a_{ar{t.b}lood}(\theta)}{\sigma(L)}}.}}} & (48)\end{matrix}$

The following mathematical definitions are used for defining featuresextracted from the normalized logarithmic (calibration or recording) PPGsignal, first derivative of the normalized logarithmic (calibration orrecording) PPG signal, the second derivative of the normalizedlogarithmic (calibration or recording) PPG signal and the Teager-Kaiserenergy operator applied on the normalized logarithmic (calibration orrecording) PPG signal. A list of possible global signal features of thecalibration PPG signal or the recording PPG signal is provided in Table6 below.

TABLE 6 Global Signal Features. Name Definition log_signal_signal_skewSkewness of the logarithmic (log.) normalized signallog_signal_signal_kurtosis Kurtosis of the log. normalized signallog_signal_signal_median Median of the log. normalized signallog_signal_kte_mean Mean the KTE on the log. normalized signallog_signal_kte_std Std of the KTE on the log. normalized signallog_signal_kte_skew Skewness of the KTE on the log. normalized signallog_signal_kte_kurtosis Kurtosis of the KTE on the log. normalizedsignal log_signal_kte_median Median of the KTE on the log. normalizedsignal log_signal_d1_mean Mean of the 1st derivative of the log.normalized signal log_signal_d1_std Std Deviation of 1st derivative ofthe log. normalized signal log_signal_d1_skew Skewness of the 1stderivative of the log. normalized signal log_signal_d1_kurtosis Kurtosisof the 1st derivative of the log. normalized signal log_signal_d1_medianMedian of the 1st derivative of the log. normalized signallog_signal_d2_mean Mean of the 2nd derivative of the log. normalizedsignal log_signal_d2_std Std Deviation of 2nd derivative of the log.normalized signal log_signal_d2_skew Skewness of the 2nd derivative ofthe log. normalized signal log_signal_d2_kurtosis Kurtosis of the 2ndderivative of the log. normalized signal log_signal_d2_median Median ofthe 2nd derivative of the log. normalized signal

The set of features of a blood pressure calibration model, such as modelM_(j), can include pulse related (or pulse-specific) features of pulsesof the calibration PPG signal and/or pulse related (or pulse-specific)features of pulses of the recording PPG signal. Pulse related featurescan include pulse features described above in sub-section D.3. Thecomputer system 100 or the feature extraction module 3702 can divide aPPG signal into single pulses and compute or extract respective pulsefeatures, as described in sub-section D.3 above. Besides the pulsefeatures of each pulse described in sub-section D.3, the featureextraction module 3702 can determine a quality metric of the pulse basedon the autocorrelation method described in Section E. The computersystem 100 or the feature extraction module 3702 can determine aweighted mean and a weighted standard deviation (across various pulsesegments) for each pulse. The computer system 100 or the featureextraction module 3702 can use the quality metric as a weight.

The set of features of a blood pressure calibration model, such as modelM_(j), can include demographic features of the user of the computersystem (e.g., the individual for whom blood pressure is measured).Example demographic features are listed below in Table 7.

TABLE 7 Demographic Features. Name Definition age age of the subject inyear (year of recording − year of birth) height the height of thesubject in cm weight the weight of the subject in kg sex the gender atbirth of the subject (male/female) BMI The Body mass index of thesubject

Once the calibration model is identified at STEP 3806, the featureextraction module 3704 can extract or determine the set of features forthe selected calibration model 3900 (those used by the calibration modelas input), and provide the features values to the calibration model3900. As discussed above, the set of features can include one or morecalibration references 3902, one or more calibration estimates 3904(provided by OBPM based on calibration PPG signal), one or more globalcalibration signal features 3906 (of calibration PPG signal), one ormore calibration pulse related features 3908 (of calibration PPGsignal), one or more recording estimates 3910 (provided by OBPM based onrecording PPG signal), one or more global recording signal features 3912(of recording PPG signal), one or more recording pulse related features3914 (of recording PPG signal), one or more demographic features 3916 orany combination thereof. The calibration model 3900 can use the featurevalues to determine blood pressure value(s) (e.g., systolic and/ordiastolic).

Each calibration model 3900 can be built using a subset of a trainingset of features. The calibration models 3900 can be trained, usinglabeled data, to determine the subset of features and the correspondingweighting coefficients for use to determine blood pressure measurements.During the training process, the features' selection for each model canbe performed using lasso algorithm or univariate feature selection. Oncethe model is trained, the final model to be used can be a regressionmodel (e.g., linear regression model), RandomForest regressor, SupportVector machine regressor, Multi-layer perceptron or a model of othertype that makes use of (or is defined in terms of) the selected subsetof features during the training process. Different models, such as the12 models in Table 4, can be different in terms of the correspondingsubset of selected features, in terms of the model parameters (e.g.,coefficients) and/or in terms of the model type.

G. Calibration-Free Improvement of Accuracy of Blood PressureMeasurement

Referring to FIG. 40 , a flowchart illustrating a method 4000 forcalibration free estimation of blood pressure. In brief overview, themethod 4000 can include obtaining a PPG signal (STEP 4002), andidentifying a calibration free blood pressure estimation model based onthe PPG signal (STEP 4004). The method 4000 can include generating ablood pressure value based using the identified calibration free bloodpressure estimation model and the PPG signal (STEP 4008).

FIG. 41 shows a block diagram illustrating a calibration free bloodpressure estimation model 4100, according to example embodiments. Theestimation model 4100 is also referring to herein as a smart offsetcalibration model 3900. The calibration model 3900 can include a linearregression model, a nonlinear regression model, a RandomForestregressor, a support vector machine regressor, a multi-layer perceptronor a neural network among others. Unlike the calibration model 3900, thecalibration free blood pressure estimation model 4100 does not make useof calibration data.

The method 4000 can include the computer system 100 obtaining arecording PPG signal (or new PPG in FIG. 41 ) generated from a sequenceof image frames obtained from the photodetector 110 (STEP 4002). Thesequence of image frames can be acquired by the computer system 100 formeasuring the blood pressure of the user. The computer system 100 cangenerate the recording PPG signal using the sequence of image frames asdiscussed above with regard to FIGS. 4 and 7 . Thecalibration/estimation module 516 can receive the recording PPG signalas input recording data for measuring the blood pressure of the user.

The method 3800 can include the computer system 100 or thecalibration/estimation module 516 identifying a calibration freeestimation model from a plurality of calibration free estimation modelsbased on the recording PPG signal (STEP 4004). The computer system 100or the calibration/estimation module 516 can maintain or have access tothe plurality of calibration free estimation models. The plurality ofcalibration free estimation models can include one or more linearregression models, one or more nonlinear regression models, one or moreRandomForest regressors, one or more support vector machine regressors,one or more multi-layer perceptrons, one or more neural networks or acombination thereof among others. The plurality of calibration freeblood pressure estimation models can machine learning models trainedusing labeled data.

The computer system 100 or the calibration/estimation module 516 canidentify or select the calibration model, from the plurality of bloodpressure calibration models, based on outputs of the OBPM module 3702when fed with recording PPG signal as input. The OBPM module 3702 canreceive a PPG signal (raw PPG signal) or a logarithmic PPG signal asinput. A logarithmic PPG signal is a PPG signal to which a logarithmicfunction or transformation is applied. The computer system 100 or theOBPM module 3702 can generate, from the recording PPG signal, alogarithmic recording PPG signal. The OBPM module 3702 can separatelyuse each of the recording PPG signal and the logarithmic recording PPGsignal as input and estimate (or attempt to estimate) a correspondingblood pressure measurement for each of these input signals. However, theOBPM module 3702 may not be successful in estimating a correspondingblood pressure measurement for each of these input signals.

The computer system 100 or the calibration/estimation module 516 candetermine (i) whether the recording PPG signal generates a first bloodpressure estimate and (ii) whether the logarithmic recording PPG signalgenerates a second blood pressure estimate. The computer system 100 orthe calibration module 516 can identify or select the calibration freeestimation model from the plurality of calibration free estimationmodels based on the vents (i) and (ii). For instance, the computersystem 100 or the calibration module 516 can identify or select thecalibration free estimation model from 4 different blood pressurecalibration models according to Table 8 below.

TABLE 8 Calibration free model selection. Recording signal Group_raw&logGroup_raw Group_log Group_noObpm Model 1 Model 2 Model 3 Model 4

Each of the plurality calibration free estimation models can beassociated with (or defined in terms of) a corresponding set ofparameter variables (corresponding to a subset of features specific tothat model) used to determine calibrated blood pressure values. Thesubset of features of a calibration free estimation model can includesystolic and/or diastolic blood pressure estimates generated by the OBPMmodule 3702 responsive to an input signal equal to the raw recording PPGsignal or the logarithmic recording PPG signal. For instance, the set offeatures can include a systolic blood pressure estimate generated usingthe recording PPG signal as an input signal, a diastolic blood pressureestimate generated using the recording PPG signal as an input signal, asystolic blood pressure estimate generated using a logarithmic recordingPPG signal as input signal, a diastolic blood pressure estimategenerated using the logarithmic recording PPG signal as input signal ora combination thereof.

The subset of features of a calibration free estimation model caninclude global PPG signal features of the recording PPG signal. Thefeature extraction module 3704 can extract global PPG signal features ofthe recording PPG signal from the logarithmic recording PPG signal. Theglobal PPG signal features of the recording PPG signal can includefeatures from Table 6 above.

The subset of features of a calibration free estimation model, caninclude pulse related (or pulse-specific) features of pulses of therecording PPG signal. Pulse related features can include pulse featuresdescribed above in sub-section D.3. The computer system 100 or thefeature extraction module 3704 can divide a PPG signal into singlepulses and compute or extract respective pulse features, as described insub-section D.3 above. Besides the pulse features of each pulsedescribed in sub-section D.3, the feature extraction module 3704 candetermine a quality metric of the pulse based on the autocorrelationmethod described in Section E. The computer system 100 or the featureextraction module 3704 can determine a weighted mean and a weightedstandard deviation (across various pulse segments) for each pulse. Thecomputer system 100 or the feature extraction module 3704 can use thequality metric as a weight.

The set of features of a blood pressure calibration model, can includedemographic features of the user of the computer system (e.g., theindividual for whom blood pressure is measured). Example demographicfeatures are listed above in Table 7.

Once the calibration model is identified at STEP 4004, the featureextraction module 3704 can extract or determine the subset of featuresfor the selected calibration free estimation model 4100 (those used bythe calibration free estimation model as input), and provide the featurevalues to the calibration free estimation model 4100. As discussedabove, the subset of features can include one or more recordingestimates 4102 (provided by OBPM based on recording PPG signal), one ormore global recording signal features 4104 (of recording PPG signal),one or more recording pulse related features 4106 (of recording PPGsignal), one or more demographic features 4108 or combination thereof.The calibration model 4100 can use the feature values to determine bloodpressure value(s) 4110 (e.g., systolic and/or diastolic).

Each calibration free estimation model 4100 can be built using a subsetof a training set of features. Each calibration free estimation model4100 can be trained, using labeled data, to determine the subset offeatures and the corresponding weighting coefficients for use todetermine blood pressure measurements. During the training process, thefeatures' selection for each model can be performed using lassoalgorithm or univariate feature selection. Once the model is trained,the final model to be used can be a regression model (e.g., linearregression model), RandomForest regressor, Support Vector machineregressor, Multi-layer perceptron or a model of other type that makesuse of (or is defined in terms of) the selected subset of featuresduring the training process. Different models, such as the 4 models inTable 8, can be different in terms of the corresponding subset ofselected features, in terms of the model parameters (e.g., coefficients)and/or in terms of the model type.

H. Feature-Based Classification of User Blood Pressure

Referring to FIG. 42 , a flowchart of a blood pressure classificationmethod 4200 is shown, according to example embodiments. The method 4200can include obtaining one or more PPG signals, each of which generatedfrom a corresponding sequence of images obtained from a photodetector(STEP 4202). The PPG signals can be generated as discussed above withregard to FIGS. 4 and 7 above.

The method 4200 can include extracting a plurality of features of thePPG signal (STEP 4204). The computer system 100 or the featureextraction module 3704 can extract features of the PPG signal that areused by the classification model to classify the blood pressure of thesubject. In general, the categories of features used by theclassification model are similar to the feature categories used by thecalibration free estimation model 4100. That is, the classificationmodel can use OBPM out features for the PPG signals, global signalfeatures of the PPG signals and pulse related features of the PPGsignals. The features extraction module can extract these features fromeach of the recorded (or generated PPG signals).

The features can include systolic and/or diastolic blood pressureestimates generated by the OBPM module 3702 for each of the recordingPPG signals and/or the logarithmic recording PPG signals. For instance,the set of features can include a systolic blood pressure estimate foreach recording PPG signal, a diastolic blood pressure estimate for eachrecording PPG signal, a systolic blood pressure estimate for eachlogarithmic recording PPG signal, and a diastolic blood pressureestimate for each logarithmic recording PPG signal. In someimplementations, the features can include an average of systolic bloodpressure estimate for the recording PPG signals, an average of diastolicblood pressure estimate for the recording PPG signals, an average ofsystolic blood pressure estimate for the logarithmic recording PPGsignals, and an average of diastolic blood pressure estimates for thelogarithmic recording PPG signals.

The features can include global PPG signal features of the recording PPGsignal or averages (over obtained PPG signals) thereof. The featureextraction module 3704 can extract global PPG signal features from thelogarithmic recording PPG signals. The global PPG signal features of caninclude features from Table 6 above. The feature extraction module canaverage each global signal feature for the obtained PPG signals.

The features can include pulse related (or pulse-specific) features ofpulses of the recording PPG signals or averages thereof. Pulse relatedfeatures can include pulse features described above in sub-section D.3.The computer system 100 or the feature extraction module 3704 can dividea PPG signal into single pulses and compute or extract respective pulsefeatures, as described in sub-section D.3 above. Besides the pulsefeatures of each pulse described in sub-section D.3, the featureextraction module 3704 can determine a quality metric of the pulse basedon the autocorrelation method described in Section E. The computersystem 100 or the feature extraction module 3704 can determine aweighted mean and a weighted standard deviation (across various pulsesegments) for each pulse feature. The computer system 100 or the featureextraction module 3704 can use the quality metric as a weight. Thecomputer system 100 or the feature extraction module 3704 can furtheraverage each pulse feature across the obtained PPG signals.

The feature extraction module 3704 may further provide demographicfeatures of the user of the computer system (e.g., the individual forwhom blood pressure is measured) to the classification model. Exampledemographic features are listed above in Table 7.

The blood pressure classification model can classify the blood pressureof the user based on the features received from the feature extractionmodule 3704 and internal parameters (e.g., coefficients) of theclassification model. The blood pressure classification can include“Normal” “High Blood Pressure” and/or “Low Blood Pressure”. The numberand the types of classifications can vary depending how theclassification model is designed and trained. The classification modelcan be a machine leaning model trained using labeled data. Theclassification model can be trained to receive feature averages (overmultiple generated PPG signals) as input.

I. Standalone Blood Pressure Measurement Device

Referring to FIGS. 43A-H, example illustrations of a device 4300 formeasuring the vital signs of a user is shown in different views,according to inventive concepts of this disclosure. For example, FIGS.43A-B can include perspective views of the device 4300, FIGS. 43C-F caninclude the side views of the device 4300, FIG. 43G can include the topview (sometimes referred to as a bird-eye view) of the device 4300, andFIG. 43H can include a bottom view of the device 4300.

The device 4300 can include one or more processors and at least onememory to perform the features and functionalities discussed herein. Thedevice 4300 can be a standalone or self-contained device. The device4300 can measure important vital signs (e.g., pulse rate, bloodpressure, respiratory rate, oxygen saturation, or temperature) byanalyzing a body part (e.g., a finger) of the user. The device 4300 canbe referred to as other descriptive terms, such as a node, a measurementcomponent, or a PPG analyzer, for example. The device 4300 can measureand provide the user with the result of the vital signs within a timeperiod or duration (e.g., under 30 seconds, 20 seconds, etc.). In somecases, the time period can refer to the duration for capturingmeasurements of the body part to obtain the results, which can bepredetermined by the administrator of the software of the device 4300and configurable via an update. The device 4300 can perform one or morefeatures and functionalities similarly to the computer system 100, aremote server, or a combination of remote devices to processmeasurements of the body part of the user, such as in conjunction withat least FIGS. 1-26 . In some cases, the device 4300 can performfeatures or functionalities different from the computer system 100 orother remote devices. The device 4300 can perform steps of one or moremethods discussed herein in conjunction with FIGS. 4, 12, 16 , etc. Insome cases, the device 4300 can perform a portion of the features orfunctionalities and delegate the remaining portions to at least oneremote device (e.g., the computer system 100 or a server). The device4300 can interface with one or more remote devices, such as a laptop, amobile device, a desktop computer, a server device, among others.

The device 4300 can measure one or more vital signs including at least,but not limited to blood pressure (“BP”) (e.g., systolic, diastolic, ormean arterial pressure (“MAP”)), heart rate (“HR”) (sometimes referredto as pulse rate, e.g., heart rate variability), respiratory rate (e.g.,recurrent respiratory papillomatosis (“RRp”) or breaths per minute),oxygen (“O2”) (e.g., blood oxygen saturation), and temperature (e.g.,skin temperature or body temperature). The device 4300 can beconstructed using or composed of hardware, software, or a combination ofhardware and software capable of measuring at least the vital signsmentioned above. The device 4300 can be constructed with other sensorsor components discussed herein to perform other measurements oranalyses. The device 4300 can perform error checking procedures, forexample, to raise one or more error conditions to the user in order toobtain good PPG signals for computing or generating good results basedon the PPG signals. The different error conditions can be based onreadings of the PPG signal, the temperature of the user, accelerationdata, among others.

The device 4300 can be a cuffless optical device for measuring vitalsigns of the user, e.g., via contact with a body part of the user. Thedevice 4300 can capture PPG signals by the reflection of light at thefingertip. The device 4300 can convert the PPG signal into the user'svital signs (e.g., BP, HR, RRp, or O2) by using one or more formulas oralgorithms. For example, a PPG is an optically obtained plethysmogramthat can be used to detect blood volume changes in the microvascular bedof tissue. The device 4300 can obtain PPG signals by passing lightthrough the tissues at a body part. By going through the tissue, lightcan be absorbed in some proportions by the different layers of thetissues, such as the venous blood layer, the non-pulsatile component ofthe artery blood, or the pulsatile component of artery blood. As anexample, FIGS. 3A-D can illustrate a decomposition of the absorptioncaused by the incoming light. Most of the tissues are fixed in thefinger, while the blood volume can vary, which can be measured by thedevice 4300. Therefore, the device 4300 can be used to measure thepulsatile component of the PPG signal.

The device 4300 can include a housing 4302 encapsulating one or morecomponents. The housing 4302 can refer to a casing, a shell, a frame, oran exterior enclosure for the components. The housing 4302 can includeat least one or more features, structures, elements, or appearances asdiscussed herein. The housing 4302 can include, but is not limited to,one or more light indicators 4304, a housing edge 4306, a center region4308, and a contact region 4310. The contact region 4310 can include oneor more components to measure or assist the user with capturing imagesof a body part, such as the fingertip of the user to obtain at least onePPG signal. The contact region 4310 can be referred to as or usedinterchangeably with other descriptive terms, such as a placementregion/area, a groove region, a finger region, a measurement region, orgenerally as region 4310. The region 4310 can include at least a lightpipe 4312, a groove 4314, a bottom region 4316 (e.g., a general bottomarea or region of the groove 4314, which can be part of the groove4314), a portion 4318 of the bottom region 4314, and an edge 4320 aroundthe portion 4318 of the bottom region 4314.

The housing 4302 of the device 4300 can be constructed with anymaterials including, but not limited to, plastic, steel, carbon fiber,wood, cloth, composite materials, among others. The housing 4302 can bein any shape, such as cylindrical, cubic, spherical, or a combination ofone or more shapes. As an example herein, the housing 4302 can be in acircular, cylindrical, or semi-spherical shape as shown in exampleillustrations of at least FIGS. 43A-B. In some cases, the housing 4302can exhibit a dome-like structure. In some other cases, the top of thehousing 4302 can be a flat surface. The housing 4302 can include an edge4306 surrounding (or encircling) the housing 4302. The housing 4302 mayinclude other appearance or design. The bottom of the housing 4302 caninclude a flat surface. In some cases, at least a portion of the bottomof the housing 4302 can include one or more indentation, footing, amongother elements to assist with placing the device 4300 onto a surface.

The housing 4302 can encapsulate the internal hardware component of thedevice 4300. In some cases, the housing 4302 can refer to the tophousing of the device 4300. In this case, the device 4300 can include aside 4324, a bottom housing 4326, and a foot 4328 of the device 4300.The housing 4302, the side 4324, the bottom housing 4326, and the foot4328 of the device 4300 can be referred to as the casing or externalenclosures of hardware components of the device 4300. The components ofthe external enclosures (e.g., the housing 4302, the side 4324, thebottom housing 4326, and the foot 4328 of the device 4300) can becomposed of the same materials. In some cases, the external enclosuresmay not be composed of the same materials. For example, the top housingand the bottom housing 4326 may be composed of glass and plastic, theside 4324 may be composed of aluminum, and the foot 4328 may be composedof rubber. Other materials can be used to construct the externalenclosures.

The side 4324 may be referred to as the corner or the edge of the device4300. For example, the diameter of the device 4300 can be measured fromone side to the opposite (or the other) side of the side 4324. The side4324 can be located, positioned, or sandwiched between the top housingand the bottom housing 4326. In some cases, the side 4324 or thematerial of the side 4324 may be a part of the bottom region 4316 of thegroove 4314. For example, one of the side 4324 positioned at the region4310 can be extended towards the curved end 4322 of the groove 4314.

The bottom of the housing 4302 can include curvatures, flat surfaces,among other characteristics. In edge or corners of the bottom housing4326 can be aligned with the side 4324 of the device 4300. The edge ofthe bottom housing 4326 may also be aligned with the edges of the tophousing. In some cases, the edge of the bottom housing 4326 may not bealigned with the edges of the top housing. For example, the edge of thetop housing (e.g., housing 4302) may not extend to or be flushed withall sides 4324 of the device 4300. Instead, at least an edge of the tophousing may be spaced from the side 4324. In some cases, the edges ofthe top housing can be flushed on all sides 4324 of the device 4300. Thebottom housing 4326 can include texture, pattern, or indentation tofacilitate the user handling or carrying the device 4300.

The foot 4328 can be a part of the bottom housing 4326. In some cases,the foot 4328 can be a separate component for installation to the device4300 (e.g., at the bottom housing 4326). The foot 4328 may be referredto as the bottom surface of the device 4300, bottom support, gripper,anti-slip feet, among other elements. The foot 4328 can facilitate orassist with the placement of the device 4300 onto a surface (e.g., adesk, floor, etc.). For example, the foot 4328 can be constructed withanti-slip materials, such that the device does not move when takingmeasurements. In another example, the foot 4328 can be constructed witha magnet to attach to metal objects. In some embodiments, the foot 4328can be constructed with a flexible or bendable material, such as toconform with the shapes, textures, or contours of the placement surface.In some implementations, the housing 4302 can include all components ofthe external enclosures, such as the side 4324, the bottom housing 4326,and the foot 4328 of the device 4300.

The groove 4314 can be an area within or on the housing 4302 of thedevice 4300 configured for placement of the finger of a user. The groove4314 can extend from the edge 4306 of the housing 4302 towards thecenter region 4308 (e.g., the center of the device 4300 from thetop-view perspective). The groove 4314 can include a curved end 4322towards the center region 4308 of the housing 4302. The groove 4314 caninclude one or more dimensions, such as a first dimension (e.g., length)between 20 mm and 70 mm, a second dimension (e.g., width) between 18 mmand 35 mm, and a third dimension (e.g., depth) between 3 mm and 10 mm.The dimensions of the groove 4314 can vary based on the construction,including different ranges either shorter or longer than theabove-described ranges. For example, the first dimension of the groove4314 can be between 14 mm and 80 mm, the second dimension can be between15 mm and 45 mm, and the third dimension can be between 2 mm and 12 mm.the curved end 4322 can refer to a curved region of the groove 4314,edging or curving towards the center region 4308 of the housing 4302.The center region 4308 can refer to the center portion, the center area,the midpoint of the housing 4302, or a general area of the housing 4302equidistant from the edge 4306 of the housing 4302. The center region4308 can a point at the center of the housing 4302 or a general areaportion around the center point, such as a span of 1 mm, 2 mm, or 3 mmfrom the center point.

The groove 4314 can include a bottom region 4316 which may includecurvature similar to the groove 4314. In some cases, the bottom region4316 can be or include a flat surface. The bottom region 4316 caninclude a portion 4318 composed with one or more sensors, e.g., forcapturing images of the user's finger for conversion to PPG signal. Theportion 4318 can be constructed with glass or other transparentmaterials which can allow light to pass and images outside of thehousing 4302 to be captured. The one or more sensors within the groove4314 can include a light source (e.g., similar to light source 108), aphotodetector (e.g., optical sensor or a camera), a temperature sensor,pressure sensor, among other sensors disposed within the housing 4302 tomeasure the vital signs as discussed herein. The device 4300 can includeother sensors, such as an accelerometer, tilt sensor, sound sensor,humidity sensor, etc.

The light source can be positioned at the bottom region 4316 of thegroove 4314, such as at the portion 4318 or other portions of the bottomregion 4316 or the groove 4314. The light source can be configured toemit light responsive to receiving an instruction from the one or moreprocessors of the device 4300. For example, the light source can emitlight upon an indication of a finger contacting or within the proximityof the groove 4314 or the bottom region 4316 of the groove 4314. Thedevice 4300 (or the computer system 100) can include a RGB light source,a full light spectrum light source or a combination of a RGB lightsource and an infrared (IR) light source. A full light spectrum can emitlight having a full light frequency spectrum. Light having full lightfrequency spectrum when reflected from a finger or other body part canbe used to generate PPG signals as well as to detect the amount ofoxygen in the finger or the other body part. Specifically, the IRcomponent of the full light frequency spectrum can be used to detect theamount of oxygen content in the blood of the subject. In someimplementations, the device 4300 can include a RGB light source to emitRGB light for use to generate PPG signals, and an IR light source toemit IR light for use to detect the amount of oxygen in the subject'sblood.

The photodetector can be disposed within the housing 4302. Thephotodetector can be positioned at the bottom region 4316 of the groove4314, similar to the light source. The photodetector can be adjacent tothe light source, such that the finger, when placed in the groove 4314,can cover both the photodetector and the light source simultaneously.The photodetector can capture a sequence of image frames includingimages of the finger or other body parts. The photodetector can capturea sequence of image frames while the light is emitted by the lightsource or responsive to the emission of light (e.g., immediately orshortly after the light is emitted on or into the finger). Hence, thephotodetector and the light source can operate in conjunction with oneanother to capture images of the user's finger for generating the PPGsignals. In some cases, the photodetector or the light source canoperate in conjunction with other sensors, such as a temperature sensor(e.g., to measure the temperature of the finger while capturing theimages) or an accelerometer (e.g., to measure the acceleration of thedevice 4300 during capturing of the images).

The device 4300 (or the computer system 100) can include a RGBphotodetector, a full light spectrum photodetector, or a combination ofa RGB photodetector and an infrared (IR) photodetector. The device 4300can include a full light spectrum photodetector capable of detecting thefull frequency spectrum of light. The device 4300 can include lightfilters to filter RGB light and IR light from light detected by the fulllight spectrum photodetector. In some implementations, the device 4300can include a separate RGB photodetector to detect RGB light and aseparate IR photodetector to detect IR light. Detected RGB light can beused to generate PPG signals, and detected IR light can be used bemeasure the amount of blood oxygen of the subject.

The device 4300 can include one or more processors configured togenerate a PPG signal of a user using the sequence of image frames todetermine the blood pressure or other vital signs of the user. Thedevice 4300 can obtain the PPG signal by the reflection of light, e.g.,via capturing images of the reflection of light from the fingerindicative of decomposition of the PPG signal. The features orfunctionalities for the device 4300 to obtain the PPG signal can bedescribed in further detail in at least FIGS. 3A-D, for example. Thedevice 4300 can include other components for processing the datacaptured by one or more sensors composed in the housing 4302, forexample. The device 4300 can include at least a memory or a data storagefor storing images, processed data, among other information generated bythe device 4300 or received from one or more remote servers or devices.The one or more processors of the device 4300 can be in electricalcommunication with other components within the device 4300. The one ormore processors can transmit instructions to the components. Theinstructions can trigger at least a feature or a function of therespective components, e.g., turning the light source on or off, dimmingthe light, increasing light intensity, capture an image, record a video(capturing a sequence of images), among other commands. The one or moreprocessors can perform other features or functionalities discussedherein.

The housing 4302 can include at least one light pipe 4312. The lightpipe 4312 may be referred to as a light rail, a light strip, atranslucent perimeter with lights, or a luminescent thread. The lightpipe 4312 can be disposed within the housing 4302. In some cases, thelight pipe 4312 can be attached or coupled to the housing 4302. Thelight pipe 4312 can be located at the upper edge of the groove 4314(sometimes referred to as an indented channel into the housing 4302 ofthe device 4300). The light pipe 4312 can extend across the upper edgeof the groove 4314, such as from a first edge portion of the housing4302, to the center region 4308, and to a second edge portion of thehousing 4302. The light pipe 4312 can include curvatures or bendaccording to the design of the groove 4314. In some cases, the lightpipe 4312 can be integrated with the groove 4314, such that the lightpipe 4312 is a part of the groove 4314. In some cases, the light pipe4312 can be inside the groove 4314, such as between the upper edge ofthe groove 4314 and the bottom region 4316 of the groove 4314. In someimplementations, the light pipe 4312 (or an additional light pipe) canbe implemented at the bottom region 4316 of the groove 4314.

The light pipe 4312 can receive instructions from the one or moreprocessors of the device 4300 configured to turn on or off (e.g.,illuminate or darken). For example, the light pipe 4312 can illuminateor emit lights to prompt the user to take the measurement of the one ormore vital signs. The light pipe 4312 can illuminate indicating orguiding the placement of the finger. In this case, by illuminatingaround the groove 4314, the light pipe 4312 can assist the user with anindication of a general area to place the finger, even in a darkenvironment. Therefore, the light pipe 4312 can allow the user to betteridentify the groove 4314 (e.g., surrounded by the light pipe 4312) orthe location of where to place the finger for measuring the vital signs.

The light pipe 4312 can be configured to prompt the user at apredetermined time to measure at least one vital sign. The predeterminedtime (sometimes referred to as a reminder time) can be configured by theuser, such as every day, week, bi-week, or month at a certain hour,minute, or second in the day. In some cases, the reminder time can beconfigured to other times during the day or night. The light pipe 4312,along with other lighting components (e.g., light source or lightindicator 4304), can emit any colored lights (e.g., RGB light) at anysupported intensity level or lumens. Further, the light pipe 4312 andother lighting components can pulsate at one or more frequencies orintervals based on the configuration of the device 4300. In some cases,the light pipe 4312 and the light indicator 4304 can emit lightsynchronously. For example, the light pipe 4312 and light indicator 4304can blink, flash, or pulsate at the same interval. Flashing the lightscan indicate at least one of triggering of an error condition, an alert,or a reminder to measure vital signs. The frequency or interval of thelighting pulse can be increase or decrease based on user configurationor configuration by a remote server managing the software or firmware ofthe device 4300, for example.

The device 4300 can include one or more tactile surfaces as depicted inFIGS. 46A-46D. The tactile surface can include or can be at least one ofa rough surface, a protrusion, an indentation, a ridge, a smoothtexture, a hard texture, a soft texture, among other types of featuresor textures. The tactile surface can be different from the texture ofbetween one or more portions of the bottom region 4316. For example, aportion 4318 of the bottom region 4316 can include a tactile surface(e.g., at least a type of tactile surface can occupy the portion 4318 ofthe bottom region 4316). The tactile surface can be a rough texture,while other portions of the bottom region 4316 can be a smooth texture.The tactile surface can assist or aid the user to position the fingersuch that the pulp region (e.g., fleshy mass on the palmar aspect) ofthe finger is positioned above at least the photodetector, e.g., byproviding users with a tactile feedback when in contact with the tactilesurface. For example, the user can physically sense (e.g., feel bytouching) the tactile surface to determine the position of the fingerwith respect to at least one of the portions of the bottom region 4316.

In some cases, the tactile surface can be at least a part of the portion4318 or at least a part of the portion 4318 can include the tactilesurface. In some other cases, the tactile surface can be present at theportion 4318, which can be different from the surface features of thebottom region 4316. For example, the portion 4318 can include thetactile surface surrounding one or more translucent areas where thephotodetector and the light source are positioned. In this case, if theportion 4318 is constructed with glass, the area or position above thephotodetector and the light source can be composed of clear glasseswhile rough glasses surround the clear glasses, for example. In somecases, the tactile surface can include a protrusion 4320 at the bottomregion 4316. The protrusion 4320 can form a closed loop to accommodatethe pulp region of the finger of the user. The protrusion 4320 canencapsulate the portion 4318 of the bottom region 4316. In some cases,the protrusion 4320 can be a ridge, a channel, or an indent forming aclosed loop at the bottom region 4316 of the housing 4302. Theprotrusion 4320 (or the indent) can include a thickness (or depth) ofless than 1 mm. The thickness or depth can be configured during themanufacturing process. The protrusion 4320 can be constructed with otherthicknesses, such as above at least 1 mm.

The photodetector can be positioned within the portion 4318 of thebottom region 4316. The photodetector can be configured to captureimages of the pulp region of the finger of the user. The photodetectorcan be initiated to capture the images upon detecting the finger withinthe proximity or in contact with a portion of the groove 4316, forexample. The photodetector can be an RGB photodetector to capture imageswith various colors.

The light source can be positioned within the portion 4318 of the bottomregion 4316. For example, the portion 4318 can position the light sourcewith the photodetector. The light source can be adjacent to or withinthe proximity of the photodetector. The light source can be spaced fromthe photodetector between, for example, 2 mm to 5 mm, among otherdistances where neither the photodetector nor the light source extendsbeyond the portion 4318 of the bottom region 4316. The light source canbe spaced at a distance such that the light can penetrate through theflesh layers and to the pulp region of the finger. Accordingly, thephotodetector can capture the reflection of the lights emitted from thelight source into the finger.

The portion 4318 of the bottom region 4316 can include a first dimension(e.g., width) between 2 mm and 10 mm and a second dimension (e.g.,length) between 5 mm and 20 mm. The length and width can beinterchangeable. The portion 4318, as well as other components orfeatures of the device 4300, can include other dimensions, e.g., thefirst dimension may be between 1.9 mm to 12 mm and the second dimensionmay be between 4 mm and 25 mm. The portion 4318 can be distanced fromthe center of the curved end 4322 of the groove 4314 (e.g., measured atany point of the portion 4318 to the curved end 4322). For example, thedistance can be greater than 2 mm and less than 15 mm from the center ofthe curved end 4322. The distance can be less than or equal to 2 mm orgreater than or equal to 15 mm, in some cases. The distance of theportion 4318 to the curved end 4322 can be similar to a distance fromthe portion 4318 to the center region 4308 of the housing 4302, in somecases.

The groove 4314 (and the light pipe 4312) can extend towards the centerregion 4308 of the housing 4302. In some implementations, the groove4314 can extend to the midpoint of the center region 4308. In someimplementations, the groove 4314 can extend near the midpoint of thecenter region 4308. In some implementations, the groove 4314 can extendbeyond the midpoint of the center region 4308, based on theconfiguration or construction of the device 4300. The bottom region 4316of the groove 4314 can be a leveled surface. In some implementations,the bottom region 4316 can be angled (e.g., inclined or declined) fromthe leveled surface.

The device 4300 can include a communication interface composed withinthe housing 4302. The communication interface can be Bluetooth, Wi-Fi,radio frequency (“RF”) communication, LTE, among other communicationports. The communication interface can be configured to establish one ormore communication channels with remote devices or remote servers. Thecommunication interface can communicate, transmit, or receive datapackets to or from other remote devices. The one or more processors ofthe device 4300 can cause the communication interface to transmit datato a remote device. For example, the one or more processors can causethe communication interface to transmit the PPG signal to a remotedevice or a remote server. In this example, the device 4300 can delegateone or more tasks to the remote device or remote server. The task caninclude determining a measurement of one or more vital signs of the userusing the PPG signal. Further, the device 4300 can delegate the task ofdisplaying the measurement of one or more vital signs to the remotedevice (e.g., for display on the display device 112 of the computersystem 100). In some cases, the one or more processors of the device4300 can perform the tasks delegated to the remote device.

In some implementations, the device 4300 can include a display devicewith one or more features or functionalities similar to the displaydevice 112. The display device can be disposed within the housing 4302.The display device can be located or positioned opposite to the region4310. For example, if the region 4310 is at the south end of the device4300, the display device can be located at the north end of the device4300. The display device can face the side where the region 4310 ispositioned, such that characters can be upright for comprehensibility bythe user. The display device can provide a graphical user interface(“GUI”) to the user. The display device can display information relatedto the device 4300, such as the results from the measurement includingone or more vital signs, the body temperature or finger temperature ofthe user if the device 4300 is equipped with a temperature sensor, anerror condition that is raised or triggered by the one or moreprocessors, a notification from a remote device, status of the device4300 including update status, connection type, connection quality,battery percentage, or connected remote devices, or other measurementresults. In some cases, the display device can display other informationavailable on a remote server or within the memory or configuration ofthe device 4300, such as the time, the temperature of the room, weatherforecast, alarm function, among others. In some cases, the displaydevice can provide the user with hints on how to improve the quality ofthe PPG signal including, but not limited to, warming the finger if theuser has a cold finger, readjusting the position of the finger (e.g.,direction and distance to adjust the finger), maintaining stability ornot to move while taking measurements, place the finger into the groove4314 or the portion 4318 of the grove 4314, among other hints orinstructions. Accordingly, using the display device, the device 4300 candisplay one or more vital signs (e.g., a blood pressure) for the user,determined using the PPG signal.

In some implementations, the device 4300 can include at least one of apressure sensor, a thermometer, or an oximeter for assisting withdetermining one or more vital signs of the user. The sensor may beposition at the portion 4318 of the bottom region 4316 with thephotodetector or the light source, adjacent to the portion 4318, otherportions of the bottom region 4316, or at a portion of the groove 4314.For example, the device 4300 can use the pressure sensor to measure thepressure applied by the finger to a portion 4318 of the bottom region4316 of the groove 4314. Based on the saturation level (e.g., lowsaturation or high saturation), the device 4300 can notify the user toincrease or decrease the pressure applied by the finger. In someembodiments, the device 4300 can use the thermometer (e.g., skintemperature probe) to measure the temperature of the finger. Based onthe temperature of the finger (e.g., too cold), the device 4300 cannotify the user to warm up the finger for improving the quality of thesignal. In another example, the device 4300 can use the oximeter tomeasure the oxygen level in the blood flowing through the finger. Thedevice 4300 can notify the user of the oxygen level, e.g., if the oxygenlevel is normal, too high, or too low. The device 4300 can notify theuser via the display device disposed within the housing 4302.

In some implementations, the device 4300 can include a visual outputdevice. The visual output device can be different from a display device.The visual output device can include one or more light indicators 4304.Each of the light indicators 4304 can be shaped like a dot, a square, alight strip, or other shapes of any size to fit within the housing 4302.The one or more light indicators 4304 can be disposed at any positionwithin the housing 4302, such as offset to the left of the contactregion 4310 and the center region 4308, offset to the right of thecontact region 4310 and the center region 4308, adjacent to the contactregion 4310, adjacent to a display device of the device 4300, or otherportions of the housing 4302. The visual output device can provide anindication of the quality of the sequence of image frames to the user.For example, the device 4300 (e.g., the one or more processors) canassess the quality of the sequence of image frames captured by thephotodetector. The device 4300 can assess the quality of the PPG signalgenerated based on the captured sequence of image frames. The device4300 can select a light output from various light outputs based on thequality of the sequence of image frames or the quality of the PPGsignal. The light outputs can include instructions for differentlighting effects to be performed by the visual output device. The lightoutputs can include various configurations or animations for the visualoutput device. The device 4300 can cause the visual output device toemit at least one light output (e.g., the light output selected based onthe quality of the image frames or the PPG signal). The light output canbe referred to as a light signal. The light signal can be indicative ofthe quality of the sequence of image frames or the quality of the PPGsignal. Emitting the light output can refer to instructing one or morelight indicators 4304 to emit lights.

In some embodiments, the light indicators 4304 can emit lights based onthe light signal. The light indicators 4304 can emit any light color,such as red, green, or blue. In view of the example illustrations of atleast FIG. 43A, the device 4300 can include four light indicators 4304on the left and another four light indicators 4304 on the right of thedevice 4300. The device 4300 can include other numbers of lightindicators 4304, such as a total of 3, 6, 10, 12, etc. In someimplementations, only one side of the device 4300 may include the lightindicators 4304. As referred to in this example for simplification, thedevice 4304 can include a total of four light indicators 4304 disposedof in the housing 4302. An example animation and sequences of the lightindicators 4304 can be as follows. When the user placed a finger in thegroove 4314 or channel, one or more light indicators 4304 can pulserapidly three times (e.g., three times within one second or half asecond) indicating that the device 4300 detected the finger. After thedevice 4300 detects the finger, the light indicators 4304 can representa quality meter indicating the quality of the sequence of images or thePPG signal. The light indicators 4304 can perform an animation while thesignal quality is being measured. The animation can stop once the signalquality has been established. Further, the light indicators 4304 canchange color (or animation) at any processing or analysis stage (e.g.,during the evaluation of the signal quality, transition from evaluatingthe signal quality to displaying the quality of the signal, etc.).

Once the quality level or metric is available, the device 4300 caninstruct the visual display device to emit at least one of various lightindicators 4304 indicating the quality of the signal. For example, inthe case of four light indicators 4304, emission of one light indicator4304 can represent low quality, with increasing quality up to four-lightindicators 4304 representing high quality. The light indicators 4304 mayflash while displaying the quality level or when completing a task, forexample. During the measurement or recording of the sequence of imageframes, the device 4300 can refresh or update the number of lightindicators 4304 emitting lights, based on the variation of the signalquality.

In some implementations, the device 4300 can alert or raise an errorcondition to the user if the quality meter or metric is below athreshold. For example, if the device 4300 includes four lightindicators 4304, the threshold may be set to 3 lights. If the quality ofthe signal declines to 1 to 2 lights, the device 4300 can transmit aninstruction to the light indicators 4304 to change from blue to orangecolor, thereby indicating that the signal quality is not good. In somecases, the device 4300 can change the color indicating an errorcondition (e.g., poor quality signal) based on the quality meter fallingbelow the threshold for a predetermined duration. For example, if thepredetermined duration is configured to 2 seconds, the device 4300 maynot change the color of the light indicators 4304 falling below 3 lightsuntil after 2 seconds.

In some embodiments, the device 4300 can be configured with a timeoutthreshold indicating a duration which the user should improve thequality meter before the device 4300 time out. The thresholds can beconfigured by the administrator of the device 4300 during manufacturingor as part of the software, which can be updated via over-the-airupdates, data packets from a remote server, or a physical connection toa remote device to download the updated software, for example. Furtherfrom the previous example, the device 4300 can change the color of thelight indicators 4304 from orange to red if the quality metric does notimprove within the timeout threshold. The light indicators 4304 can alsoblink or flash rapidly for a predetermined number of times (e.g., 3, 5,or 7 times). Responsive to triggering the timeout, the device 4300 canstop capturing images and wait for the user to restart the measurement.To restart the measurement, the user can lift the finger off the groove4314 for at least 1 second (or other pre-configured duration) and placethe finger back in the groove 4314, for example. In some embodiments,the device 4300 can include a button for initiating or restarting themeasurement. In some implementations, the initiate or restart ameasurement, the user can use an application from a remote device torestart the measurement using the device 4300.

The device 4300 can be configured to execute any animation (e.g., lightpulse, breathing effect, dimming effect, wave effect, etc.), brightnesslevel, or color (e.g., R, G, or B) for the light indicators 4304 basedon the status or stage of the one or more processors. For example,different animations, color, or brightness can be initiated during theprocess of detecting the finger, when taking images of the finger,generating the PPG signal based on the images, computing the qualitymetric or level, displaying the quality level, measuring the vitalsigns, or when the results of the vital signs are available, eitherdisplayed on the local display device of the device 4300 or transmittedto an application executing on a remote device of the user (e.g.,application 114 executing on the computer system 100). In someimplementations, if the device 4300 includes a display device, thedevice 4300 may not be composed of the visual output device in additionto the display device. Instead, in these implementations, the displaydevice can perform display an animation, dotted lights, or otherfeatures similar to the light indicators 4304.

In some implementations, the device 4300 can detect if the finger of theuser is positioned correctly on the photodetector and/or the lightsource positioned at a portion 4318 of the bottom region 4316 of thegroove 4314. For example, the device 4300 can determine at least thelocal variation values of the pixels for each image of the sequence ofimages captured by the photodetector. The local variation values of therespective pixels can represent the variation magnitude (e.g., based onintensity or color values) for each of the inner pixels of thedownsampled image compared to pixels orthogonally adjacent to therespective pixel. Based on the local variation values, the device 4300can determine a respective metric for each downsampled image. Further,for each pixel of downsampled images, the device 4300 can determine arespective aggregate pixel similarity score used to determine a vectorindicative of a position of the user's finger. The vector can representthe distance of the center of mass of the finger (e.g., the pulp regionof the finger or other regions used to apply pressure to the portion4318 of the bottom region 4316) from the center of the photodetector.The vector can indicate the quality of the sequence of image frames orthe PPG signal, where the shorter the distance can indicate higherquality, and the longer the distance can indicate poorer quality of theimages or the signal. Thus, the device 4300 can determine if the fingeris present near the photodetector and the direction (e.g., based on thevector) the user should reposition the finger to be centered above thephotodetector for good quality measurement. The methods and techniquesto determine the adjustment of the position of the finger can bedescribed in conjunction with at least FIGS. 16-18C, for example.Accordingly, the device 4300 can instruct the user to adjust theplacement of the finger within the groove 4314, responsive todetermining a poor quality of the sequence of image frames or poorquality of the PPG signal.

In some implementations, the device 4300 may perform a “preflight test”routine, sometimes referred to as an initial test or booting process.The initial test can be performed on startup (e.g., during first bootfor each day or weekly). The device 4300 can determine the quality ofone or more sensors (e.g., photodetector, light source, or temperatureprobe) or signals received by the sensors. The device 4300 mayrecalibrate one or more sensors based on the initial test. For example,the device 4300 can determine that the images captured by thephotodetector are blurred before detecting a finger in the groove 4314.Based on this determination, the device 4300 can recalibrate or resetthe settings of the photodetector to obtain clear images. The device4300 can perform other startup processing or checks to prepare formeasuring one or more vital signs of the user. In some implementations,the initial test can include or refer to a process for detecting thatthe finger is present in the groove 4314 and the conditions (e.g., coldfinger or temperature condition, saturation condition, among others) forstarting the measurements are met, prior to measuring the vital signs.

In some implementations, the device 4300 can compose of differentfeatures or components based on the model, version, or constructedcomponents of the device 4300. For example, each model of the device4300 can compose of different sensors, microcontroller unit (“MCU”),connectivity, or display capabilities (e.g., with or without a displayunit local to the device 4300). The MCU may process raw data, includingat least the PPG signal or blood pressure recording data locally on thedevice 4300. With a display device, the device 4300 can display at leastthe blood pressure value (e.g., systolic/diastolic) and heart ratelocally using the embedded display device. In some cases, the MCU mayprocess sensor data and display all vital measurements using theembedded display device to the user. Hence, based on the respectivemodel, the device 4300 can include additional features for measuringvital signs or be limited to various features for measuring some of thesubsets of vital signs, for example.

The one or more hardware components of the device 4300 can be describedin further detail as follows. For example, the hardware components caninclude at least a core sensor, temperature sensor (e.g., temperatureprobe), embedded display (e.g., display device), battery, pressuresensor (e.g., a piezoelectric sensor), haptic feedback, finger groovelight pipe (e.g., light pipe 4312), cloud connectivity, Bluetooth,printed circuit board (“PCB”), and sound generator (e.g., speaker). Thecore sensor can include a series of RGB LEDs and a photodetector (e.g.,specialized or RGB) configured to capture the PPG signal. The device4300 can use the temperature sensor to measure the body part temperature(e.g., skin temperature of the finger) for estimating the temperature ofthe user's body. The device 4300 can use the embedded display to displaymeasurement information (e.g., BP or other vitals), as well as otherilluminated indicators, such as alerts, hints, error conditions, fingerposition, etc. The embedded display can include an LED array that shinesthrough the surface of the housing 4302, for example. The device 4300can include a battery to power the device 4300 when not plugged in orcharging. The battery can be composed of a rechargeable li-ion battery,for example. The device 4300 can include components for enabling sleepor hibernation mode, such that the device 4300 can be woken up upon atouch of the finger or having the finger near the device forconservation of energy. The device can use a photoelectric sensor,proximity sensor, or pressure sensor to determine when the user places afinger in the groove or on a portion of the device 4300. In some cases,the device 4300 may not be woken up from hibernation, unless a certainamount of force is applied to the sensor area, e.g., by the finger.

In some embodiments, the pressure sensor can be a piezoelectric sensor.The device 4300 can use the pressure sensor to measure or indicate anadjustment (e.g., via a display device either of the device 4300 or aremote device) to the amount of force that the finger is applying to thesensor area (e.g., the portion 4318 of the bottom region 4316). Thedevice 4300 can use the haptic feedback (e.g., haptic actuator orvibration motor) to communicate different state-events with the user,such as if the measurement is starting, the measurement ended, etc. Thelight pipe 4312 can illuminate through the housing 4302 from the channelsupporting the light pipe 4312. The light pipe 4312 can be used to guidethe user to place the finger into the groove 4314. The light pipe 4312can receive a programmable alert schedule (e.g., from an applicationexecuting on a remote device) for a time of day or week to prompt analarm to provide a reminder to take measurements. The device 4300 canuse cloud connectivity, which can be a low-powered Wi-Fi connection, totransmit data to a remote server or a cloud.

In some embodiments, the device 4300 can use Bluetooth communication,for example, to sync the BP results and user's vital information with anapplication executing on a remote device and/or stream (e.g.,concurrently or asynchronously) the raw PPG signal or information to theapplication for data analysis. The one or more components of the device4300 can be in electrical communication with the PCB, which can carry ortransmit internal communication between components (e.g., battery powerstatus, wireless communication, display data, etc.) of the device 4300.The device 4300 can use the speaker to provide a sound, such as an alarmsound or notification to the user. The generated sound can bepreconfigured by the administrator of the application synced to thedevice 4300 or by factory default. The generated sounds can provide anaudio notification for any texts or characters display on the displaydevice of either the device 4300 or a remote device connected to thedevice 4300. In some cases, the speaker can notify the measurementresults, among other information available to the device 4300, to theuser. In some implementations, the device 4300 can include a microphoneto receive voice commands, such as turn on/off the device 4300, “tell methe measurements,” start measurements, among others. In someimplementations, the UI displayed by the display device of the device4300 can be replicated to the application of a remote device, or viceversa where the display from the remote device can be synced to thedisplay of the device 4300.

In some implementations, the device 4300 may be display-less. Forexample, the device 4300 may not include a display device. Instead, thedevice 4300 can be connected to a remote device executing an application(e.g., application 114). For example, the device 4300 can transfer atleast one of captured images, PPG signals, raw PPG or BP recording data,processed data, one or more error conditions, or among other data oralerts to be present to the user via a display device of a remotedevice. In some cases, the device 4300 can transfer the data to a webserver or a portion using the communication interface. In someimplementations, the results (e.g., vital signs or health condition),error conditions, or hints to reposition the finger can be displayed ona UI within an application executing on the remote device. In this case,the user can view or obtain at least the measurement values inside theapplication of the remote device or a portion within a web browser.

In some embodiments, the device 4300 can transmit the processed data tothe connected remote device for displaying one or more vital signs ormeasurement results via the UI of the application to the user. In somecases, the device 4300 may be a relay device to provide data to theremote device for processing. For example, the device 4300 can collectone or more images of the finger. Responsive to capturing the images,the device 4300 can transmit the image data to the remote device fordownsampling, converting the images to a PPG signal, among otherprocessing techniques. In this case, the remote device can determine oneor more vital signs of the user and display the respective vital signson a display device of the remote device for the user. Since the device4300 can be easily wiped or cleaned, the device 4300 can be sharedacross multiple users for measuring vital signs. Therefore, with asingle device 4300, multiple users can measure their respective vitalsigns to determine their health conditions. Accordingly, the device 4300can be used routinely to maintain the health conditions of any user.

In some implementations, the device 4300 (or the computer system 100)can include or can host one or more batteries to charge the device 4300(or the computer system 100). The device 4300 (or the computer system100) may include one or more coils (e.g., located at the bottom surfaceof the device 4300) to wirelessly charge the device 4300 (or thecomputer system 100). The device 4300 (or the computer system 100) mayinclude a micro USB port that can be used to charge the device 4300 (orthe computer system 100). The device 4300 (or the computer system 100)may include other means for wired or wireless charging of the device4300 (or the computer system 100).

In some implementations, the device 4300 (or the computer system 100)can include fingerprint-based authentication means or capabilities. Forinstance, the photodetector 110 can capture images of a subjectfingerprints, and the processor 102 can process the captured fingerprintimages, and compare them to stored fingerprints of a user of the device4300 (or the computer system 100). The device 4300 (or the computersystem 100) can authenticate users based on the comparison of capturedfingerprint images and maintained fingerprints of one or more registeredusers. In some implementations, the device 4300 (or the computer system100) can include a separate photo sensor for receiving images of userfingerprints.

J. Design of Standalone Blood Pressure Measurement Device

FIG. 44A-H illustrate different views of a device 4400 for measuringvital signs of a user, according to example embodiments. FIG. 45A-Hillustrate different views of a device 4500 for measuring vital signs ofa user in broken lines, according to example embodiments. FIG. 46A-D4600 illustrate perspective and top views of a device with a roughsurface in a groove, according to example embodiments.

The ornamental design which is claimed is shown in solid lines in thedrawings. The broken line showing a device, a housing, a groove, lightindicators, a bottom region, or other components of the device, in thedrawings are for illustrative purposes only and form no part of theclaimed design. Broken lines formed by equal length dashes showunclaimed portions of the design.

While the accompanying drawings illustrate one or more exemplaryembodiments, it should be understood that according to other exemplaryembodiments that should be considered to be within the possession of theinventors of the present application at the time this application isbeing filed, it is contemplated that any illustrated solid lines (orportions thereof) may be converted to broken lines and that anyillustrated broken lines (or portions thereof) may be converted to solidlines so as to claim or disclaim portions, components, or sub-componentsof the designs shown. It is further contemplated that shading may beadded or removed to claim or disclaim the corresponding surfaces.

Each method described in this disclosure can be carried out by computercode instructions stored on computer-readable medium. The computer codeinstructions, when executed by one or more processors of a computingdevice, can cause the computing device to perform that method.

While the disclosure has been particularly shown and described withreference to specific embodiments, it should be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the inventiondescribed in this disclosure.

While this disclosure contains many specific embodiment details, theseshould not be construed as limitations on the scope of any inventions orof what may be claimed, but rather as descriptions of features specificto particular embodiments of particular inventions. Certain featuresdescribed in this specification in the context of separate embodimentscan also be implemented in combination in a single embodiment.Conversely, various features described in the context of a singleembodiment can also be implemented in multiple embodiments separately orin any suitable subcombination. Moreover, although features may bedescribed above as acting in certain combinations and even initiallyclaimed as such, one or more features from a claimed combination can insome cases be excised from the combination, and the claimed combinationmay be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated in a single software product or packaged intomultiple software products.

References to “or” may be construed as inclusive so that any termsdescribed using “or” may indicate any of a single, more than one, andall of the described terms.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain embodiments, multitasking and parallel processingmay be advantageous.

What is claimed is:
 1. A device for measuring blood pressure comprising:a housing including a curved top portion and a flat base portion, thecurved top portion including at least one visual output device includinga plurality of light emitting diodes (LEDs), the curved top portionincluding a groove to receive a finger of a user, the groove having acurved end towards a center of the housing, the groove partially definedby a bottom surface; a light source disposed within the housing andpositioned beneath the bottom surface of the groove and configured toemit light in the groove; a photodetector disposed within the housingand positioned beneath the bottom surface of the groove and configuredto capture a sequence of image frames while the light is emitted; one ormore tactile surface features extending from the bottom surface of thegroove and defining a first portion of the bottom surface of the grooveto aid the user in positioning a pulp region of the finger above thephotodetector, wherein the photodetector is positioned within the firstportion and configured to capture images of the pulp region of thefinger of the user; and a processor configured to generate aphotoplethysmographic (PPG) signal of a user using the sequence of imageframes to determine a blood pressure of the user, and wherein theprocessor is configured to control the plurality of LEDs to i) provide afirst indication indicating that a finger of the user has been detected;and ii) provide a second indication indicating a quality of the PPGsignal.
 2. The device of claim 1, wherein the groove extends from anedge of the housing towards a center of the device.
 3. The device ofclaim 1, wherein the one or more tactile surface features include aprotrusion at the bottom surface and forming a closed loop toaccommodate the pulp region of the finger of the user.
 4. The device ofclaim 3, wherein the thickness of the protrusion is less than 1 mm. 5.The device of claim 1, wherein the first portion of the bottom surfacehas a first dimension between 2 mm and 10 mm and a second dimensionbetween 5 mm and 20 mm.
 6. The device of claim 1, wherein the firstportion of the bottom surface of the groove is less than 15 mm andgreater than 2 mm from a center of the curved end.
 7. The device ofclaim 1, wherein the photodetector includes at least one of: ared-green-blue (RGB) photodetector; a full light spectrum photodetector;or a combination of a RGB photodetector and an infrared photodetector.8. The device of claim 1, further comprising a communications interfaceconfigured to communicate with a remote device.
 9. The device of claim8, wherein the processor is configured to: cause the communicationinterface to transmit the PPG signal to the remote device, the remotedevice configured to: determine a blood pressure measurement of the userusing the PPG signal; and display the blood pressure measurement on adisplay device.
 10. The device of claim 1, further comprising a displaydevice disposed within the housing, the processor is configured to:determine a blood pressure measurement of the user using the PPG signal;and display the blood pressure measurement on a display device.
 11. Thedevice of claim 1, further comprising at least one of: a pressure sensorto measure pressure applied by the finger to a first portion of thebottom region of the groove; a thermometer to measure a temperature ofthe finger; or an oximeter to measure oxygen level in blood flowingthrough the finger.
 12. The device of claim 1, wherein the processor isconfigured to: assess a quality of the sequence of image frames or aquality of the PPG signal; select, based on the quality of the sequenceof image frames or the quality of the PPG signal, from the plurality ofLEDs, a subset of the plurality of LEDs; and cause the subset of theplurality of LEDs to emit a light output.
 13. The device of claim 1,further comprising a light pipe disposed within the housing at an upperedge of the groove, the processor configured to cause the light pipe tobe illuminated to prompt the user to take a measurement of the one ormore vital signs.
 14. A method of measuring vital signs of a user,comprising: receiving a finger within a groove of a housing of a device,the groove partially defined by a bottom surface, the device including:a curved top portion and a flat base portion, the curved top portionincluding at least one visual output device including a plurality oflight emitting diodes (LEDs), the curved top portion including thegroove; a processor; a light source disposed within the housing andpositioned beneath the bottom surface of the groove; a photodetectordisposed within the housing and positioned beneath the bottom surface ofthe groove; and one or more tactile surface features extending from thebottom surface of the groove and defining a first portion of the bottomsurface of the groove to aid the user in positioning a pulp region ofthe finger above the photodetector, wherein the photodetector ispositioned within the first portion and configured to capture images ofthe pulp region of the finger of the user; emitting, by the lightsource, light in the groove; capturing, by the photodetector, a sequenceof images of the finger while the light is emitted; and generating, bythe processor, a photoplethysmographic (PPG) signal using the sequenceof images to determine one or more vital signs of the user, and whereinthe processor is configured to control the plurality of LEDs to i)provide a first indication indicating that a finger of the user has beendetected; and ii) provide a second indication indicating a quality ofthe PPG signal.
 15. The method of claim 14, further comprising:determining a measurement of the one or more vital signs using the PPGsignal; and displaying the measurement of the one or more vital signs ona display device.
 16. The method of claim 14, further comprising:transmitting the PPG signal to a remote device, the remote devicedetermining a measurement of the one or more vital signs using the PPGsignal and displaying the measurement of the one or more vital signs ona display device.
 17. The method of claim 14, further comprising:assessing, by the processor, a quality of the sequence of image framesor a quality of the PPG signal; and providing, via the plurality ofLEDs, a light signal indicative of the quality of the sequence of imageframes or the quality of the PPG signal.
 18. The method of claim 17,further comprising: instructing the user to adjust placement of thefinger within the groove, responsive to determining a poor quality ofthe sequence of image frames or a poor quality of the PPG signal. 19.The method of claim 14, further comprising providing, via a fingergroove light pipe located at an upper edge of the groove, a light signalto prompt the user to take a measurement of the one or more vital signs.20. The method of claim 14, wherein the one or more tactile surfacefeatures include a protrusion at the bottom surface and forming a closedloop to accommodate the pulp region of the finger of the user.